Venue: NAACL 2025

# PromptOptMe: Error-Aware Prompt Compression for LLM-based MT Evaluation Metrics

Daniil Larionov1,2 Steffen Eger1,3

1 NLLG, 2 University of Mannheim, 3 University of Technology Nuremberg danii.larionov@uni-mannheim.de

###### Abstract

Evaluating the quality of machine-generated natural language content is a challenging task in Natural Language Processing (NLP). Recently, large language models (LLMs) like GPT-4 have been employed for this purpose, but they are computationally expensive due to the extensive token usage required by complex evaluation prompts. In this paper, we propose a prompt optimization approach that uses a smaller, fine-tuned language model to compress input data for evaluation prompt, thus reducing token usage and computational cost when using larger LLMs for downstream evaluation. Our method involves a two-stage fine-tuning process: supervised fine-tuning followed by preference optimization to refine the model's outputs based on human preferences. We focus on Machine Translation (MT) evaluation and utilize the GEMBA-MQM metric as a starting point. Our results show a \(2.37{\times}\) reduction in token usage without any loss in evaluation quality. This work makes state-of-the-art LLM-based metrics like GEMBA-MQM more cost-effective and efficient, enhancing their accessibility for broader use.

## 1 Introduction

The rapid advancement of Natural Language Generation (NLG) technologies has led to an increasing reliance on automated systems for producing human-like text across various domains, including machine translation (MT). As these systems become more prevalent, the need for effective and efficient evaluation metrics to assess generated content quality has also grown. Evaluating NLG systems is a fundamental challenge in the field of Natural Language Processing (NLP).

Traditionally, automatic evaluation metrics such as BLEU Papineni et al. (2002), ROUGE Lin (2004), BERTScore Zhang et al. (2019), and MoverScore Zhao et al. (2019) have been widely used due to their simplicity and ease of implementation. BLEU and ROUGE measure n-gram overlap between the generated text and reference texts, but they often fail to capture the semantic meaning and penalize legitimate lexical variations Callison-Burch et al. (2006). BERTScore and MoverScore leverage contextual embeddings from pre-trained language models to compute similarity at a deeper semantic level; however, they can still struggle with capturing nuanced errors in meaning and may not effectively evaluate aspects like factual correctness or language fluency Freitag et al. (2021a); Kocmi et al. (2021); Chen and Eger (2023).

Subsequently, trained evaluation metrics such as COMET Rei et al. (2022), BLEURT Sellam et al. (2020), and xCOMET Guerreiro et al. (2023) emerged. These models are trained on human-annotated datasets to predict quality scores, allowing them to better align with human judgments. Despite their improved quality, these metrics require substantial amounts of labeled data

Figure 1: The two-stage model training approach used in PromptOptMe. At the first stage, the model is fine-tuned in a supervised way to adapt it for the compression task and prompt format. At the second stage, we utilize preference data obtained through evaluation of compressed prompts to train the model to select the best compression for each example.



for training and may not generalize well across different tasks or domains.

With the advent of large language models (LLMs) like GPT-4 [1], researchers have started to explore their potential for NLG evaluation through prompting instead of traditional training. Metrics such as GEMBA-DA [23], GEMBA-MQM [50], AutoMQM [69], and G-Eval [78] utilize LLMs by providing instructions and few-shot examples within prompts. This approach offers several key benefits: **a)** elimination of task-specific training, as LLMs can perform evaluations based on prompts without the need for labeled datasets; **b)** high-quality results, with LLM-based metrics demonstrating superior alignment with human judgments and more effectively capturing linguistic nuances and contextual information; **c)** flexibility and generalization, since these models can be easily adapted to different tasks and domains by modifying prompts, offering greater flexibility compared to traditional trained metrics.

Despite these advantages, using LLMs like GPT-4 for evaluation is computationally expensive due to the extensive token usage required by elaborate prompts. For instance, the original GEMBA-MQM prompt typically requires between 1100 and 1200 tokens per example, accounting for exhaustive instructions and three few-shot examples. If we aim to evaluate this metric on 60k examples from the WMT22 Metrics Challenge Test Set [13], the total token usage would be on the order of:

Total Tokens \(\approx\) 60k \(\times\) 1200 \(\approx\) 72M tokens.

Given that GPT-4 pricing is $10 per million tokens used, the estimated cost for evaluating the entire dataset would be:

Cost per full run \(\approx\)\(\frac{72\text{M}}{1\text{M}}\)\(\times\) 10 \(\approx\)\(\$720\).

These substantial costs present a barrier to practical deployment, especially in budget-constrained settings. It also makes LLM-based metrics like GEMBA-MQM egregiously expensive for large-scale evaluation scenarios, like online re-ranking of MT-systems outputs or web-scale dataset processing [23].

To address these challenges, we introduce **PromptOptMe**,1 a novel approach to prompt optimization that reduces token usage and computational costs in LLM-based evaluation by utilizing a smaller, fine-tuned language model to compress input data without sacrificing necessary information. We summarize our contributions as follows: **a)** We propose a two-stage fine-tuning process for **PromptOptMe**, involving supervised fine-tuning and preference optimization [14], to refine the model's outputs based on actual metric behavior with compressed inputs. **b)** We apply **PromptOptMe** to MT, achieving a \(2.32\times\) reduction in token usage for MT evaluation, without any loss in evaluation quality.

Footnote 1: PromptOptME stands for “**Prompt Optimization for Metrics”** and is inspired by PrExMe [13], a recent method to investigate prompt exploration for MT evaluation metrics.

Our work enhances the accessibility of advanced LLM-based evaluation metrics by making them more cost-effective and efficient for broader use in real-world NLG applications, thereby promoting diversity and inclusion in NLP research and application by enabling participation from under-resourced communities [1].

## 2 Related Work

The field of prompt optimization has seen substantial activity in recent years. This section covers works related to prompt optimization, efficient evaluation, and explainable metrics, highlighting how our approach builds upon and differs from existing research.

We begin by examining the landscape of **explainable evaluation metrics**[13, 14, 15]. Leiter et al. (2024) propose a taxonomy for such metrics, within which the GEMBA-MQM metric falls into the category of fine-grained error metrics. Naturally, these metrics critically depends on preservation of error-spans within source text and translation. Unlike xCOMET [15], GEMBA-MOM's sentence-level scores are derived solely from extracted errors and their severities, adhering to the MQM guidelines [14]. Due to this fact, we designed our method to fine-tune the model to preserve the error-spans in the source and translation text by utilizing the WMT MQMM annotated dataset. Further, PromptOptMe can 

also be seen as explaining prompt-based evaluation metrics because it yields additional insights into which prompt parts are relevant for a metric to perform with high-quality.

In the realm of **prompt compression and optimization**, several notable approaches have been developed. LLMLingua Jiang et al. (2023) introduces a coarse-to-fine prompt compression method, employing a budget controller and an iterative token-level compression algorithm. This approach utilizes a smaller language model to compute token importance, substantially reducing token usage and inference latency across various tasks. Building upon this foundation, LLMLingua-2 Pan et al. (2024) proposes a data distillation procedure to derive knowledge from LLMs for efficient and faithful task-agnostic prompt compression. By formulating prompt compression as a token classification problem and leveraging a Transformer encoder, LLMLingua-2 further improves compression efficiency and generalizability. Unlike both of those approaches, we specifically target the task of LLM-based evaluation metrics. We incorporate preference optimization based on actual metric behavior, tailoring the prompt compression to preserve essential evaluation information.

Black-Box Prompt Optimization (BPO) Cheng et al. (2024) applies prompt optimization for black-box LLM alignment, using a prompt preference optimizer trained on human preference data. This model-agnostic approach enhances the alignment of LLM outputs with human intents without requiring access to model parameters or additional training. Our method shares similarities in leveraging preference learning, but we extract these preferences by exploiting measurable quality differences during prompt execution with optimized outputs.

PRewrite Kong et al. (2024) proposes an automated prompt engineering approach using reinforcement learning to rewrite prompts for improved prediction quality. While both PRewrite and our method optimize prompts without modifying the underlying language model, our approach distinguishes itself by simultaneously striving to reduce prompt size and improve quality.

In the domain of **efficient MT evaluation metrics**, several approaches have been proposed to address the computational challenges posed by large trained metrics. xCOMET-lite Larionov et al. (2024), COMETinho Rei et al. (2022b), and FrugalScore Kamal Eddine et al. (2022) all employ techniques such as pruning, quantization, and distillation to create smaller, more efficient models. While these methods achieve impressive compression ratios, resulting in models with under 500M parameters, they often experience a drop in correlation with human judgment. Our approach aims to mitigate this quality loss by continuing to rely on large backbone LLMs while improving efficiency through targeted prompt optimization.

Larionov et al. (2023) explore an alternative approach to creating an efficient version of BERTScore and MoverScore by replacing the underlying encoder model with smaller alternatives, such as pruned or distilled models. We touch this subject too, by evaluating our prompt compression model not only on large backbone LLM GPT-do, but also on smaller models: GPT-4o mini and LLaMa 3.2. As we demonstrate in Section 5, our approach proves effective across different model sizes, improving evaluation efficiency even on smaller LLMs.

## 3 Method

In this section, we detail our two-stage approach for prompt optimization aimed at reducing token usage and computational costs in LLM-based evaluation metrics. We first look into the optimization of prompt inputs (source texts and machine translations) and later detail our approach to compressing the rest of the prompt. The first stage involves supervised fine-tuning of a language model to learn the specifics of the prompt compression task. The second stage employs preference optimization using the Odds-Ratio Preference Optimization (ORPO) algorithm Hong et al. (2024) to refine the model's outputs based on the observed behavior of the metric. We chose this algorithm due to its specific focus on reducing the likelihood of generating the rejected responses, which are, in our case, low-quality compressed source and translation texts. The overall approach is visualized in Figure 1.

### Stage One: Supervised Fine-Tuning

In the first stage, we perform supervised fine-tuning of a language model to enable it to effectively compress input texts while preserving essential information required for accurate evaluation. In the context of the MT evaluation task, the model is trained to accept original uncompressed source 

texts and their respective machine translations and to generate three outputs:

1. A compression rate \(r\), selected as a floating-point number from the set \({\cal R}_{comp}=\left\{0.3,0.4,0.5,0.6,0.7,0.8,0.9,1.0\right\}\), where \(1.0\) indicates no compression.
2. A list of potential substrings from the source and MT texts that contain translation errors.
3. The compressed versions of the source and MT texts.

We achieve two primary objectives by prompting the model to extract potential error spans from the source and MT texts. First, this process teaches the model to actively search for translation errors, thereby enhancing its understanding of MT evaluation. Second, by conditioning the compression of the texts on these identified error spans, the model ensures that critical information related to translation errors is preserved in the compressed outputs. This approach allows the compressed texts to retain essential details necessary for accurate evaluation, even with reduced token usage.

To construct the training dataset for supervised fine-tuning, we utilize Multidimensional Quality Metrics (MQM)-annotated data from the WMT Metrics shared tasks Freitag et al. (2021b). For each example in the dataset, we extract error-annotated spans based on the MQM annotations. We then perform random token removal to generate compressed texts, ensuring that the identified error spans remain intact in the compressed versions. The compression rate \(r\) is randomly selected from the predefined set for each example.

Formally, let \(S\) denote the original source text, \(T\) the original MT text, and \(E\) the set of error spans extracted from MQM annotations. The compressed source text \(S^{\prime}\) and compressed MT text \(T^{\prime}\) are generated by removing tokens from \(S\) and \(T\), respectively, while preserving all tokens within \(E\).

This supervised fine-tuning process trains the model to perform the prompt compression task effectively, balancing the reduction in token count with the retention of essential information required for accurate evaluation.

### Stage Two: Preference Optimization

In the second stage, we apply ORPO to further tune the model based on preferences between various compressions, enabling it to select the optimal compression for each example. The goal is to optimize the trade-off between token reduction and the preservation of evaluation quality.

To create the preference dataset needed for ORPO, we generate compressed versions of each example using the previously defined set of compression rates. For each compression rate \(r\), we obtain compressed texts \(S^{\prime}_{r}\) and \(T^{\prime}_{r}\) using the finetuned model from Stage One. We then incorporate these compressed texts into the original GEMBA-MQM prompt and submit them to GPT-4o to obtain evaluation scores \(s_{r}\).

The evaluation scores are compared to the score \(s_{1.0}\) obtained from the uncompressed texts (with \(r=1.0\)). For each example, we identify:

* The _chosen_ compression with rate \(r_{chosen}\) whose evaluation score \(s_{r_{chosen}}\) has the minimal absolute difference from \(s_{1.0}\). If multiple compression rates have equally minimal differences, we select the one with the lowest \(r\) to prioritize higher compression.
* The _rejected_ compression with rate \(r_{rejected}\), whose evaluation score \(s_{r_{rejected}}\) has the maximal absolute difference from \(s_{1.0}\). If multiple compression rates have equally maximal differences, we select the one with highest \(r\).

Formally, we define the absolute score difference \(\Delta_{r}=|s_{r}-s_{1.0}|\). The chosen and rejected compression rates satisfy:

\[r_{chosen} =\arg\min_{r\in{\cal R}_{comp}}\Delta_{r},\] \[r_{rejected} =\arg\max_{r\in{\cal R}_{comp}}\Delta_{r}.\]

Using these preferences, we train the model from Stage One using the ORPO algorithm, which adjusts the model's parameters to increase the likelihood of generating outputs corresponding to the chosen compression rates over the rejected ones. The ORPO loss function is defined based on the odds ratio of the probabilities assigned to the chosen and rejected outputs:

\[{\cal L}_{\text{ORPO}} =\mathbb{E}_{(x,y_{c},y_{r})}\left({\cal L}_{\text{SFF}}+\lambda \cdot{\cal L}_{\text{OR}}\right)\] \[{\cal L}_{\text{OR}} =-\log\sigma\left(\log\frac{odds_{\theta}(y_{c}|x)}{odds_{ \theta}(y_{r}|x)}\right)\] \[odds_{\theta}(y|x) =\frac{P_{\theta}(y|x)}{1-P_{\theta}(y|x)}\] 

where \(x,y\) are a prompt and a completion respectively, \(P_{\theta}(y_{c}|x)\) and \(P_{\theta}(y_{r}|x)\) are the probabilities assigned by the model with parameters \(\theta\) to the chosen and rejected outputs, respectively. \(\mathcal{L}_{SFT}\) refers to standard cross-entropy loss used for language modeling.

This preference optimization process teaches the model to select compressions that yield evaluation scores closest to those obtained with uncompressed texts, effectively maintaining evaluation quality while reducing token usage. By prioritizing compression rates that minimize score discrepancies, the model learns to balance the trade-off between efficiency and quality.

### Simplified Prompts

In the context of MT evaluation, the source texts and their translations are typically short, often comprising up to two or three sentences. However, the GEMBA-MQM prompt includes long instructions, outlining the entire MQM error typology. Those instructions are included in both few-shot examples and in the target example. We hypothesize that this surplus of tokens raises computational costs without necessarily enhancing evaluation quality.

Initially, we also trained a prompt optimization model using preference data to compress the instruction component of the prompt. However, upon evaluation, we found that the compressed instructions generated were effectively the same across different inputs. This uniformity suggested that a fixed simplified instruction could suffice without dynamic compression per example.

Moreover, as demonstrated in Section 4, the use of these simplified instructions did not adversely affect the metric quality. The evaluation quality remained comparable to that achieved with the original, more verbose GEMBA-MQM prompt.

Consequently, we adopt a fixed simplified instruction template for our evaluations and discontinued further instruction compression. For reference, both the original GEMBA-MQM prompt and our simplified version are provided in the Appendix A.

## 4 Experiments

In this section, we evaluate the effectiveness of our proposed prompt optimization approach in reducing token usage while maintaining evaluation quality in LLM-based evaluation metrics. We conduct experiments on MT evaluation and assess the compression efficiency and the impact on evaluation quality.

As a base model for fine-tuning, we utilize the LLaMA-3.2 Dubey et al. (2024) model in two variants -- 1B and 3B parameters -- chosen for its extensive vocabulary and multilingual pre-training beneficial for handling diverse language pairs.

Stage One: Supervised Fine-TuningFor the first stage, we use the WMT Metrics shared task datasets with MQM annotations from the years 2020 to 2022 Freitag et al. (2021b). We exclude the 'news' portion of WMT22 dataset for validation purposes. The remaining training data consists of approximately 145k examples across three language pairs: English-Russian, English-German, and Chinese-English.

Following the approach described in Section 3, we construct prompts with system instructions for the model. The prompt follows this template:

```
System:YouareanhelpfulAIassistantthatintelligentlycompressesandsummarizesthegivenMachineTranslationoutputsforfurtherevaluation. User:CompressthefollowingMTinputandoutput:Source::''{src-text}"AT:""{mt-text}"
```

The expected model completion is:

``` Assistant:Rate=rQuality-relevantpartsofSource:[error-span-1];...;[error-span-N]orNone Quality-relevantpartsofTranslation:[error-span-1];...;[error-span-N]orNone CompressedSource:"""{compressed-src}"' CompressedMT:""{compressed-mt}"" Here, {_src-text_} and {_mt-text_} denote the original source and machine-translated texts. We provide an example of the text compression with trained model in Figure 2, where we demonstrate both model inputs -- source text and machine translation -- as well as the outputs -- compression rate, error spans and compressed texts 

We train the model for one epoch using a batch size of 64 and a learning rate of \(2\times 10^{-5}\) with cosine warmup over 6% of the training steps. We apply a weight decay of 0.01. For efficient training, we use Low-Rank Adaptation (LoRA) adapters Hu et al. (2022), with the following parameters: rank \(r=32\), scaling factor \(\alpha=16\), and dropout rate of 0.5.

Stage Two: Preference OptimizationFor the second stage, we construct a preference dataset by selecting a subset of 20k examples from the training data. For each example, we generate compressed versions at each compression rate from our predefined set, resulting in eight different compressions per example.

We incorporate these compressed examples into the original GEMBA-MQM prompt and evaluate them using the GPT-do model to obtain evaluation scores. Based on these scores, we select the _chosen_ and _rejected_ compressions following the procedure described in Section 3.

Using this preference dataset, we fine-tune the model from Stage One employing ORPO. We train for three epochs with a batch size of 64, a learning rate of \(1\times 10^{-5}\), and cosine warmup over 6% of training steps. We set the ORPO \(\lambda\) parameter to 0.1 following the original paper authors.

Evaluation ProcedureTo evaluate the effectiveness of our model, we apply it to compress the examples in the test set. This compression is performed on both the few-shot examples and the target example within the GEMBA-MQM prompts. We then conduct MT evaluation using these compressed prompts with proprietary LLMs GPT-do and GPT-do-mini and the openly-available LLaMA-3.2-Vision model with 90B parameters. In our experiments, we utilize both the original and simplified prompts (as described in Section 3.3) and generate outputs in both plain text format and in JSON.

We assess the quality of the evaluations by computing pairwise accuracy at the system level, as suggested by Deutsch et al. (2023), comparing our results with human judgments. Additionally, we calculate segment-level Pearson \(r\) and Kendall-\(\tau\) correlation for each language pair to measure the agreement between our evaluations and human assessments. In addition to the validation set, we use WMT23 Metrics Challenge Freitag et al. (2023) dataset to perform meta-evaluation of our approaches and compare them with existing metrics.

To quantify the efficiency gains, we measure the cumulative token usage during the evaluations. This provides insights into the computational cost savings achieved through our prompt optimization approach. To compare the effectiveness of our approach, we also include two baseline results obtained using the LLMLingua-2 Pan et al. (2024) prompt compression method. In those cases, we have applied _microsoft/llmlingua-2-xlm-robertar-large-meetingbank_ to compress source texts and machine translations similarly as we do with PromptOptMe. We fix and test two compression rates: 30% and 50%.

## 5 Results

In this section, we present the results of our experiments evaluating the effectiveness of our proposed prompt optimization approach.

As shown in Table 1, the baseline model, **GPT-4o ref**, uses GPT-4o with the full (uncompressed) GEMBA-MQM prompt, ends up with a total amount of 19M input tokens used for entire test set of 16k examples and serving as a reference

Figure 2: Example of Model-generated compression. Reference translation and Gold MQM annotations are provided for readers convenience and are not part of the model prompt.

 

point. The simplified prompting approach from Section 3.3, denoted as **GPT-4o****lite**, reduces the token usage to 10.4M tokens, achieving a reduction rate of 2.04\(\times\) while maintaining a similar pairwise accuracy of 0.7736, but lower segment-level Kendall \(\tau\) correlations across the language pairs (0.3838 for En-Ru, 0.3207 for En-De, and 0.2890 for Zh-En).

By applying our prompt optimization model, PromptOptMe-3B, with **GPT-4o****lite** prompt template, we achieve the highest reduction rate of 2.37\(\times\), reducing token usage from 19M to 8.3M tokens. Surprisingly, PromptOptMe-3B attains the best segment-level Kendall \(\tau\) correlations across all language pairs, with 0.4455 for En-Ru, 0.4065 for En-De, and 0.3738 for Zh-En, fully recovering reduced scores in the baseline, while also retaining the same level of system-level pairwise accuracy.

This trend continues with other backbone LLMs. For instance, using PromptOptMe-3B with **GPT-4o****mini****lite** prompt, we, again, achieve the same reduction rate of \(2.37\times\) and observe an improvement in pairwise accuracy to **0.7842**, which is a 2.77% increase over the **GPT-4o****mini****ref** baseline. However, the segment-level correlations exhibit mixed results: while there is a substantial decrease for En-Ru (14.67% lower than the baseline), there is a positive improvement for En-De (an increase of 2.30%) and Zh-En (a 3.57% increase).

Similarly, for the **LLaMa3.2-90B** model, applying PromptOptMe-3B results in a reduction rate of \(2.27\times\). The pairwise accuracy remains the same as the baseline (0.7526), but the segment-level Kendall \(\tau\) correlations show modest improvements for En-Ru (a 2.61% increase) and En-De (9.28% higher), while Zh-En experiences a decrease of 12.68%.

Thus, overall, we see substantial efficiency gains and simultaneously often an increase in quality of the resulting metrics; however, there are cases when metric quality reduces, sometimes considerably.

In comparison with the baseline approaches with the LLMLingua-2, we see dramatically decreased quality in both compression settings, as compared to our approach and baseline metrics. For the system-level pairwise accuracy, we observe a 31%-40% decrease from the original uncompressed metric. On a segment level, the quality decrease is even more catastrophic. For two out of three language pairs, the correlation plunges to near zero. For Zh-En, it drops to 0.19-0.12. This indicates that

\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline
**Model + Prompt** & **Token Usage** & **Reduction Rate** & **Pairwise Accuracy** & **En-Ru \(\tau\)** & **En-De \(\tau\)** & **Zh-En \(\tau\)** \\ \hline GPT-4o ref & 19M & 1.00 & 0.7789 & 0.4365 & 0.3950 & 0.3692 \\ GPT-4o mini ref & 19M & 1.00 & 0.7631 & 0.3723 & 0.3165 & 0.3472 \\ LLaMa3.2-90B ref & 20M & 1.00 & 0.7526 & 0.3416 & 0.2920 & 0.3576 \\ GPT-4o lite & 10.4M & 1.84 & 0.7736 & 0.3838 & 0.3207 & 0.2890 \\ \hline GPT-4o lite & & & & & & \\ PromptOptMe-3B & 8.07M & 2.37 & 0.7736 & **0.4455** & **0.4065** & **0.3738** \\ PromptOptME-1B & 8.8M & 2.15 & 0.7644 & 0.4122 & 0.3900 & 0.3743 \\ \hline GPT-4o inline & & & & & & \\ PromptOptMe-3B & 8.07M & 2.37 & 0.7842 & 0.3177 & 0.3238 & 0.3596 \\ PromptOptME-1B & 8.8M & 2.15 & 0.7531 & 0.3089 & 0.3125 & 0.3468 \\ \hline ILLaMa3.2-90B like & & & & & & \\ PromptOptThe-3B & 8.8M & 2.27 & 0.7526 & 0.3505 & 0.3191 & 0.3123 \\ PromptOptME-1B & 9.0M & 2.22 & 0.7345 & 0.3203 & 0.2891 & 0.3000 \\ \hline GPT-4o lite & & & & & & \\ LLMLingua @ 50\% & 7.6M & 2.5 & 0.4736 & 0.0055 & 0.0492 & 0.1247 \\ LLMLingua@ 30\% & 7.0M & 2.7 & 0.5421 & 0.00 & 0.03 & 0.1949 \\ \hline \hline  & & & & & & \\ \end{tabular}
\end{table}
Table 1: Evaluation results for machine translation with different prompting strategies and models. **GPT-4o****ref** refers to the original GPT-4 with the full (uncompressed) GEMBA-MQM prompt. **lite** denotes the simplified prompting approach with JSON-formatted output. PromptOptMe-3B and PromptOptMet-1B represent our prompt optimization models based on LLAMA 3.2 with 3B and 1B parameters, respectively, used for input compression. **Token Usage** shows the total token usage. **Reduction Rate** is the token reduction rate compared to the baseline (**-ref** for each model). **Pairwise Accuracy** stands for pairwise system-level accuracy. **En-Ru**\(\tau\), **En-De**\(\tau\), and **Zh-En**\(\tau\) refer to the segment-level Kendall \(\tau\) correlations for English-Russian, English-German, and Chinese-English language pairs, respectively. Boldface indicates the best quality in each column.

 

LLMLingua-2 substantially damages the prompt to the point of complete non-usability.

In Table 2, we extend our evaluation to the full WMT23 Metrics Challenge dataset, to further assess the generalization of our prompt optimization model. The reduction rate on the WMT23 Metrics Challenge is slightly smaller (1.7\(\times\) compared to 2.37\(\times\) on the WMT22 'news' subset). This likely reflects a more cautious compression for the Hebrew-English pair, which was unseen during training. Despite this, the model preserves evaluation quality, compared to uncompressed metrics, as indicated by the Pairwise Accuracy.

## 6 Discussion

Our proposed prompt optimization model, PromptOptMe, demonstrates substantial improvements in computational efficiency for machine translation evaluation. By integrating PromptOptMe with a simplified prompting approach, we achieve substantial reductions in token usage--up to 2.37\(\times\) less than the baseline--while maintaining or even improving evaluation quality across various metrics. These findings suggest that it is possible to compress inputs substantially while maintaining evaluation quality. We speculate that outputs of few-shot examples, which are not compressed in any way in our experiments, play a crucial role in establishing model quality in MT evaluation, as they themselves cover all error severity levels as well as error categories from MQM typology. Preserving entire source and translation texts, in turn, appears to be redundant. We are able to retain quality while preserving only part of the text, which includes error-spans.

The superior quality of PromptOptMe-3B, compared to the smaller PromptOptMe-1B model, indicates that a larger prompt optimization model is more effective at compressing input data efficiently. Specifically, PromptOptMe-3B refers to our prompt optimization model based on LLaMA 3.2 with 3 billion parameters, achieves 10% (2.37 vs. 2.15) higher compression rate on average, along with 2%-3% higher pairwise accuracy, compared to PromptOptMe-1B. It demonstrates the importance of model capacity in the compression process. Further experiments with larger LLMs could potentially help get a better understanding of scaling laws for that particular task.

When applying PromptOptMe-3B to different backbone LLMs, such as **GPT-4o mini** and **LLaMa3.2-90B**, we observe varying results. For **GPT-4o mini**, we achieve an improved pairwise accuracy compared to the baseline, with a reduction rate of 2.37\(\times\). In contrast, when applying PromptOptMe-3B to **LLaMa3.2-90B**, the pairwise accuracy remains consistent with the baseline, and we observe modest improvements in segment-level Kendall \(\tau\) correlations for some language pairs. In both cases, we notice a slight degradation in segment-level correlations in one of the language pairs, however. Nonetheless, we can conclude that our prompt optimization model achieves generalization across different backbone LLMs, despite being trained only on preferences obtained through GPT-4o.

Additionally, compared to baseline approaches using LLMLingua-2, our method clearly outperforms in both system-level and segment-level evaluations. The substantial quality degradation observed with LLMLingua-2, particularly at the segment level, indicates that it may not be suitable for effective prompt compression in MT evaluation tasks. This result is in agreement with our hypothesis that preserving quality-relevant spans in text is essential for maintaining evaluation quality. LLMLingua-2, as a non-task-specific prompt compression method, is not trained to take that into account.

Evaluation on the WMT23 Metrics Challenge further supports the robustness of our prompt optimization model. The preserved pairwise accuracy and high pearson correlation indicates that our approach generalizes across different datasets and language pairs. Future work may investigate adaptive compression strategies to better handle low-resource or previously unseen language pairs.

## 7 Conclusion

In this paper, we introduced **PromptOptMe**, a prompt optimization approach designed to reduce token usage and computational costs in large language model-based evaluation metrics. By leveraging a smaller, fine-tuned language model to compress input data, we achieved substantial reductions in token usage without compromising evaluation quality. Specifically, **PromptOptMe** reduced token usage by up to \(2.37\times\) while 

maintaining or improving segment-level Kendall \(\tau\) correlations across multiple language pairs as well as system-level pairwise accuracy.

Our approach enhances the practicality of LLM-based evaluation metrics, making them more cost-effective and accessible for large-scale natural language generation (NLG) applications. By addressing the computational expense associated with extensive prompt token usage, **PromptOptMe** enables more efficient evaluations. This reduction in computational cost makes state-of-the-art MT evaluation more accessible to under-resourced researchers and students, who may have limited access to computational resources or funding for extensive LLM usage. By lowering the barriers to high-quality evaluation, **PromptOptMe** democratizes the ability to conduct advanced research and development in machine translation. Our approach is especially useful in this case, as it does not require a compromise in metric quality, offering no quality drop while decreasing the token usage. Moreover, we are able to effectively compress the inputs for smaller LLMs as well, such as GPT-4o mini and LLMa 3.2 90B, which makes SOTA evaluation metrics even more accessible.

While our experiments focused on machine translation, where error highlights and annotations are common, we acknowledge that extending this approach to other NLG tasks may present challenges, particularly in scenarios where such detailed error annotations are not available. Future work could explore adapting **PromptOptMe** to other NLG evaluation tasks, investigating how input compression affects evaluation quality in contexts without explicit error spans, and determining the generalizability of our method across various domains.

It is also important to note, that the optimized prompts generated by **PromptOptME** are not guaranteed to strictly adhere to the MQM error typology. This divergence from the standard evaluation framework may impact the consistency and interpretability of the evaluation results, especially when comparing outputs across different systems or studies. Future work should focus on enhancing the alignment of compressed prompts with established evaluation standards to improve the reliability and comparability of the assessments.

We plan to release the code and models to the public at [https://github.com/NL2G/promptoptme](https://github.com/NL2G/promptoptme) to facilitate further research and application in this area. We believe that offers a promising direction for efficient high-quality LLM-based evaluations.

## Acknowledgements

The NLLG group gratefully acknowledges support from the Federal Ministry of Education and Research (BMBF) via the research grant "Metrics4NLG" and the German Research Foundation (DFG) via the Heisenberg Grant EG 375/5-1.

## Limitations

While our proposed approach demonstrates substantial reductions in token usage without sacrificing evaluation quality, there are several limitations to this study that we acknowledge.

First, our experiments are primarily focused on machine translation MT evaluation. Although we achieved notable efficiency gains in this domain, we have not extensively tested the applicability of **PromptOptMe** on other NLG tasks. Further

\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline
**Model + Prompt** & **Token Usage** & **Reduction Rate** & **Pairwise Accuracy** & **En–De \(r\)** & **He-En \(r\)** & **Zh-En \(r\)** \\ \hline GPT-4o ref & 60.32M & 1.00 & 0.896 & 0.479 & 0.422 & 0.523 \\ GPT-4o mini ref & 60.32M & 1.00 & 0.928 & 0.410 & 0.433 & 0.458 \\ GPT-4o lite & 40.63M & 1.48 & 0.936 & 0.610 & 0.424 & 0.547 \\ GPT-4o-mini lite & 40.63M & 1.48 & 0.916 & 0.490 & 0.419 & 0.467 \\ GEMBA-MOMM[nonref] & – & – & 0.944 & 0.502 & 0.401 & 0.449 \\ \hline GPT-4o & & & & & & \\ PromptOptMe-3B & 34.81M & 1.73 & 0.944 & 0.587 & 0.453 & 0.574 \\ GPT-4o-mini & & & & & & \\ PromptOptMe-3B & 34.81M & 1.73 & 0.932 & 0.508 & 0.459 & 0.516 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Evaluation results on WMT23, computed using ‘mt-metrics-eval’ evaluation framework. En–De \(r\), He–En \(r\) and Zh–En \(r\) stand for segment-level Pearson-\(r\) correlation with human judgment for respective language pairs. ’GEMBA-MQM[nonref]’ represent results of the original submission of GEMBA-MQM to the WMT23 Metrics Challenge.

 

research is necessary to validate the effectiveness of **PromptOptMe** across a broader spectrum of NLG applications.

Second, we evaluated our method using only one openly available language model in addition to **GPT-4o**, specifically the **LLaMA-3.2** model. While the results are promising, testing our approach on a wider variety of models, including more open-source and proprietary LLMs with diverse architectures and sizes, would help establish the generalizability and robustness of **PromptOptMe**. Future work should include experiments with additional models to better understand the applicability of our prompt optimization technique.

Additionally, our approach relies on a two-stage fine-tuning process involving supervised fine-tuning and preference optimization. This process requires access to sufficient training data and computational resources for fine-tuning the smaller language model used for prompt compression. In scenarios where such resources are limited or unavailable, the practicality of our method may be constrained. Exploring alternative approaches that require less extensive fine-tuning or that leverage zero-shot or few-shot learning could mitigate this limitation. In addition to one-time training cost, practitioners using PromptOptMe should also consider inference costs for the compression model itself. Further work should take these costs into account for more fair comparison with the baseline.

Furthermore, while our prompt optimization aims to preserve essential evaluation information, there is a possibility that some fine-grained details, such as specific MQM error categories, may not be fully captured in the compressed prompts. This could potentially affect the precision in identifying certain types of translation errors, leading to less detailed evaluations. Ensuring that critical information is retained during compression without increasing token usage remains a challenge that warrants further investigation.

## References

* Achiam et al. (2013) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Alterschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. Cited by: SS5.
* Beltaadi and Eger (2023) UScore: an effective approach to fully unsupervised evaluation metrics for machine translation. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, Dubrovnik, Croatia, pp. 358-374. External Links: Link Cited by: SS6.
* C. Callison-Burch, M. Osborne, and P. Koehn (2006) Re-evaluating the role of bleu in machine translation research. In 11th Conference of the European Chapter of the Association for Computational Linguistics, Trento, Italy, pp. 249-256. External Links: Link Cited by: SS6.
* Y. Chen and S. Eger (2023) MENLI: robust evaluation metrics from natural language inference. Transactions of the Association for Computational Linguistics11, pp. 804-825. Cited by: SS6.
* J. Cheng, X. Liu, K. Zheng, P. Ke, H. Wang, Y. Dong, J. Tang, and M. Huang (2024) Black-box prompt optimization: aligning large language models without model training. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Bangkok, Thailand, pp. 3201-3219. External Links: Link Cited by: SS6.
* D. Deutsch, G. Foster, and M. Freitag (2023) TES matter: meta-evaluating modern metrics with pairwise accuracy and tie calibration. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore, pp. 12914-12929. External Links: Link Cited by: SS6.
* A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahale, A. Lateman, A. Mathur, A. Schelten, A. Yang, A. Fan, et al. (2024) The llama 3 herd of models. arXiv preprint arXiv:2407.21783. Cited by: SS6.
* P. Fernandes, D. Deutsch, M. Finkelstein, P. Riley, A. Martins, G. Neubig, A. Garg, J. Clark, M. Freitag, and O. Firat (2023) The devil is in the errors: leveraging large language models for fine-grained machine translation evaluation. In Proceedings of the Eighth Conference on Machine Translation, Singapore, pp. 1066-1083. External Links: Link Cited by: SS6.
* M. Freitag, W. Mathew, C. Lo, E. Avramidis, R. Rei, B. Thompson, T. Kocmil, F. Blain, D. Deutsch, C. Stewart, C. Zerva, S. Castilho, A. Lavie, and G. Foster (2023) Results of wmt23 metrics shared task: metrics might be guilty but references are not innocent. In Proceedings of the 

Eighth Conference on Machine Translation_, pages 578-628, Singapore. Association for Computational Linguistics.
* neural metrics are better and more robust. In _Proceedings of the Seventh Conference on Machine Translation (WMT)_, pages 46-68, Abu Dhabi, United Arab Emirates (Hybridy). Association for Computational Linguistics.
* Freitag et al. (2021b) Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, George Foster, Alon Lavie, and Ondrej Bojar. 2021b. Results of the WMT21 metrics shared task: Evaluating metrics with expert-based human evaluations on TED and news domain. In _Proceedings of the Sixth Conference on Machine Translation_, pages 733-774, Online. Association for Computational Linguistics.
* Guerreiro et al. (2023) Nuno M Guerreiro, Ricardo Rei, Daan van Stigt, Luisa Coheur, Pierre Colombo, and Andre FT Martins. 2023. xcomet: Transparent machine translation evaluation through fine-grained error detection. _arXiv preprint arXiv:2310.10482_.
* Hong et al. (2024) Jiwoo Hong, Noah Lee, and James Thorne. 2024. Reference-free monolithic preference optimization with odds ratio. _arXiv preprint arXiv:2403.07691_.
* Hu et al. (2022) Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In _The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022_. OpenReview.net.
* Jiang et al. (2023) Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2023. LLMLingua: Compressing prompts for accelerated inference of large language models. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 13358-13376, Singapore. Association for Computational Linguistics.
* Eddine et al. (2022) Moussa Kamal Eddine, Guokan Shang, Antoine Tixier, and Michalis Vazrigiannis. 2022. FrugalScore: Learning cheaper, lighter and faster evaluation metrics for automatic text generation. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1305-1318, Dublin, Ireland. Association for Computational Linguistics.
* Kaster et al. (2021) Marvin Kaster, Wei Zhao, and Steffen Eger. 2021. Global explainability of BERT-based evaluation metrics by disentangling along linguistic factors. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 8912-8925, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Kocmi and Federmann (2023a) Tom Kocmi and Christian Federmann. 2023a. GEMBA-MQM: Detecting translation quality error spans with GPT-4. In _Proceedings of the Eighth Conference on Machine Translation_, pages 768-775, Singapore. Association for Computational Linguistics.
* Kocmi and Federmann (2023b) Tom Kocmi and Christian Federmann. 2023b. Large language models are state-of-the-art evaluators of translation quality. In _Proceedings of the 24th Annual Conference of the European Association for Machine Translation_, pages 193-203, Tampere, Finland. European Association for Machine Translation.
* Kocmi et al. (2021) Tom Kocmi, Christian Federmann, Roman Grundkiewicz, Marcin Junczys-Dowmunert, Hitotkazu Matsushita, and Arul Menezes. 2021. To ship or not to ship: An extensive evaluation of automatic metrics for machine translation. In _Proceedings of the Sixth Conference on Machine Translation_, pages 478-494, Online. Association for Computational Linguistics.
* Kong et al. (2024) Weize Kong, Spurthi Hombaiah, Mingyang Zhang, Qiaozhu Mei, and Michael Bendersky. 2024. PREFwrite: Prompt rewriting with reinforcement learning. In _Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 594-601, Bangkok, Thailand. Association for Computational Linguistics.
* Larionov et al. (2023) Daniil Larionov, Jens Grunwald, Christoph Leiter, and Steffen Eger. 2023. EffEval: A comprehensive evaluation of efficiency for MT evaluation metrics. In _Findings of the Association for Computational Linguistics: EMNLP 2023_, pages 78-96, Singapore. Association for Computational Linguistics.
* Larionov et al. (2024) Daniil Larionov, Mikhail Seleznyev, Vasiliy Viskov, Alexander Panchenko, and Steffen Eger. 2024. xcomet-lite: Bridging the gap between efficiency and quality in learned mt evaluation metrics. _arXiv preprint arXiv:2406.14553_.
* Leiter and Geger (2024) Christoph Leiter and Steffen Eger. 2024. Prexme! large scale prompt exploration of open source llms for machine translation and summarization evaluation. _ArXiv_, abs/2406.18528.
* Leiter et al. (2024) Christoph Leiter, Piyawat Lervittyakumjorn, Marina Fornicheva, Wei Zhao, Yang Gao, and Steffen Eger. 2024. Towards explainable evaluation metrics for machine translation. _Journal of Machine Learning Research_, 25(75):1-49.
* Leiter et al. (2023) Christoph Leiter, Juri Opitz, Daniel Deutsch, Yang Gao, Rotem Dror, and Steffen Eger. 2023. The Eval4NLP 2023 shared task on prompting large language models as explainable metrics. In _Proceedings of the 4th Workshop on Evaluation and Comparison of NLP Systems_, pages 117-138, Bali, Indonesia. Association for Computational Linguistics.
* Lin (2004) Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In _Text Summarization Branches Out_, pages 74-81, Barcelona, Spain. Association for Computational Linguistics 

Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-eval: NLG evaluation using gpt4 with better human alignment. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 2511-2522, Singapore. Association for Computational Linguistics.
* Opitz and Frank (2021) Juri Opitz and Anette Frank. 2021. Towards a decomposable metric for explainable evaluation of text generation from AMR. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_, pages 1504-1518, Online. Association for Computational Linguistics.
* Pan et al. (2024) Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, Menglin Xia, Xufang Luo, Jue Zhang, Qingwei Lin, Victor Ruhle, Yuqing Yang, Chin-Yew Lin, H. Vicky Zhao, Lili Qiu, and Dongmei Zhang. 2024. LLMLingua-2: Data distillation for efficient and faithful task-agnostic prompt compression. In _Findings of the Association for Computational Linguistics ACL 2024_, pages 963-981, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics.
* Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
* Peter et al. (2023) Jan-Thorsten Peter, David Vilar, Daniel Deutsch, Mara Finkelstein, Juraj Juraska, and Markus Freitag. 2023. There's no data like better data: Using QE metrics for MT data filtering. In _Proceedings of the Eighth Conference on Machine Translation_, pages 561-577, Singapore. Association for Computational Linguistics.
* Rei et al. (2022b) Ricardo Rei, Jose G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and Andre F. T. Martins. 2022a. COMET-22: Unbabel-IST 2022 submission for the metrics shared task. In _Proceedings of the Seventh Conference on Machine Translation (WMT)_, pages 578-585, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
* Rei et al. (2022c) Ricardo Rei, Ana C Farinha, Jose G.C. de Souza, Pedro G. Ramos, Andre F.T. Martins, Luisa Coheur, and Alon Lavie. 2022b. Searching for COMETINHO: The little metric that could. In _Proceedings of the 23rd Annual Conference of the European Association for Machine Translation_, pages 61-70, Guent, Belgium. European Association for Machine Translation.
* Sellam et al. (2020) Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text generation. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 7881-7892, Online. Association for Computational Linguistics.
* Zhang et al. (2019) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. _arXiv preprint arXiv:1904.09675_.
* Zhao et al. (2019) Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. 2019. MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 563-578, Hong Kong, China. Association for Computational Linguistics.

## Appendix A Original and Simplified GEMBA-MQM prompt

### Additional Details

#### Package versions

We use the following versions of key software packages in our environment:

* Python OpenAI SDK: 1.42.0
* mt-metrics-eval: 0.0.3
* SciPy: 1.14.1
* LLMLingua: 0.2.2

#### Artifact Intended Use and License

To the best of our knowledge, our use of scientific artifacts, namely Llama 3.2 pretrained model(s), is consistent with intended use policy and with the license. Those are outlined in the respective model card 2.

Footnote 2: [https://huggingface.co/meta-llama/Llama-3](https://huggingface.co/meta-llama/Llama-3).

#### Total Computational Budget

Our total computational budget for supervised finetuning and preference optimization of PromptOptMe models is 186 GPU hours.

 

* [105] M. C. B. K., and R. J. S. A. Sloane. _Theoretical Computer Science Series_. Cambridge University Press, New York, 2003.
* [106] D. L. G. T. Jones. _A mathematical theory of computer science_. Oxford university press, 1987.
* [107] P. H. W. Fowler. _Computational Principles in Mathematical Sciences_. Springer-Verlag, Berlin, 1984.
* [108] E. F. Wilson. _Mathematical Foundations of Modern Physics_. John Wiley & Sons, Inc., 1987.
* [109] N. J. Williams. _Computer Methods for Theoretical Biology_. Addison Wesley, Reading, MA, 1987.
* [110] V. I. Jordan. _Introduction to the Theory of Computing_. Addison Wesley, Reading, MA, 1987.
* [111] V. I. Jordan. _An Introduction to the Theory of Computing_. Addison Wesley, Reading, MA, 1987.
* [112] V. I. Jordan. _On the Theory of Computing_. Addison Wesley, Reading, MA, 1987.
* [113] V. I. Jordan. _Convexity: An introduction to the theory of computing_. Addison Wesley, Reading, MA, 1987.

[MISSING_PAGE_POST]

 

* [10] M. C. Bader, A. K. Goyal, and R. J. Sorensen. "Theoretical evaluation of the performance of a new machine learning system for the task-specific