ALOPE: Adaptive Layer Optimization for Translation Quality Estimation using Large Language Models
Authors: Archchana Sindhujan
Shenbin Qian
Chan Chi Chun Matthew
Constantin Orasan
Diptesh Kanojia
Venue: COLM 2025 (2025)
Keywords: Quality Estimation
Machine Translation
Translation Quality
TL;DR: This paper presents ALOPE, an adaptive layer-optimization framework for LLM-based translation quality estimation, enhancing cross-lingual transfer learning through layer-wise adaptation, dynamic weighting, and multi-head regression.

Abstract: Large Language Models (LLMs) have shown remarkable performance across a wide range of natural language processing tasks. Quality Estimation (QE) for Machine Translation (MT), which assesses the quality of a source-target pair without relying on reference translations, remains a challenging cross-lingual task for LLMs. The challenges stem from the inherent limitations of existing LLM-based QE systems, which are pre-trained for causal language modelling rather than regression-specific tasks, further elevated by the presence of low-resource languages given pre-training data distribution. This paper introduces ALOPE, an adaptive layer-optimization framework designed to enhance LLM-based QE by restructuring Transformer representations through layer-wise adaptation for improved regression-based prediction. Our framework integrates low-rank adapters (LoRA) with regression task heads, leveraging selected pre-trained Transformer layers for improved cross-lingual alignment. In addition to the layer-specific adaptation, ALOPE introduces two strategies—dynamic weighting, which adaptively combines representations from multiple layers, and multi-head regression, which aggregates regression losses from multiple heads for QE. Our framework shows improvements over various existing LLM-based QE approaches. Empirical evidence suggests that intermediate Transformer layers in LLMs provide contextual representations that are more aligned with the cross-lingual nature of the QE task. We make resultant models and framework code publicly available for further research, also allowing existing LLM-based MT frameworks to be scaled with QE capabilities.

Published as a conference paper at COLM 2025
ALOPE: Adaptive Layer Optimization for Translation Quality Estimation using Large Language Models
Archchana Sindhujan Shenbin Qian
Chan Chi Chun Matthew
Constantin Oräsan & Diptesh Kanojia
Institute for People-Centred AI and
Centre for Translation Studies,
School of Computer Science and Electronic Engineering,
University of Surrey, United Kingdom
{a. sindhujan, s. qian, c.orasan, d.Kamojjaj@eurreyyaccuu mat wettewchhncc@@gail.coo
Abstract
Large Language Models (LLMs) have shown remarkable performance across wide range of natural language processing tasks. Quality Estimation (QE) for Machine Translation (MT), which assesses the quality of source-target pair without relying on reference translations, remains a challenging cross-lingual task for LLMs. The challenges stem from the inherent limitations of existing LLM-based QE systems, which are pre- trained for causal language modelling rather than regression-speeific tasks, further elevated by the presence of low-resource languages given pre- training data distribution. This paper introduces ALOPE, an adaptive layer-optimization framework designed to enhance LLM-based QE by restructuring Transformer representations through layer-wise adaptation for improved regression- based prediction. Our framework integrates low-rank adapters (LoRA) with regression task heads, leveraging selected pre-trained Transformer layers for improved cross-lingual alignment. in addition to the layer-specific adaptation, ALOPE introduces two strategies-dynanee dynamic weighting, which adaptively combines representations from multiple layers, and multi-head regression, which aggregates regression losses from multiple heads for QE, Our framework shows improvements over various existing LLM-based QE approaches. Empirical evidence suggests that intermediate Transformer layers in LLMs provide contextual representations that are more aligned with the cross-lingual nature of the QE task. We make resultant models and framework code publicly available' for further research, also allowing existing LLM-based MT frameworks to be scaled with QE capabilities.
1 Introduction
Quality Estimation (QE) allows the output of Machine Translation (MT) to be evaluated at scale without the need for reference translations (Zerva et al., 2022). MT QE can be performed at different levels, but this paper focuses segment-level evaluation for low- resource languages, predicting the Direct Assessment (DA) score (0 < x S 100) for each segment (Graham et al., 2013). The ground truth score is obtained by averaging scores assigned by multiple human annotators (>= 3) following annotation guidelines that emphasize adequacy, fluency, and cross-lingual meaning transfer within source and machine-translated output.
https://gittbb..om/surrey ALOPE

---

Published as a conference paper at COLM 2025
Although QE annotation frameworks such Multi-dimensional Quality Metrics (MQM) (Lommel et al., 2013) provide fine-grained insights into translation errors, they impose considerable cognitive burden annotators, often making them more labour- intensive. In contrast, DA scores offer a more efficient alternative for complementing such frameworks in the overall evaluation of segment-level MT quality. Furthermore, DA scores can be leveraged to guide LLMs in reasoning over translation errors and in identifying translations that require human inspection or post-editing.
While Large Language Models (LLMs) exhibit strong performance in translation evaluation when reference translations are available (Koemi & Federmann, 2023; Qian et al., 2024), assessing cross-lingual meaning transfer in a reference-less setting remains significant challenge (Mujadia al., 2023; Sindhujan et al., 2025) for LLMs. Due to the limited representation of languages in pre-training data (Zhu et al., 2024), complexity of translation errors (Sindhujan et al., 2024), and cross-lingual alignment within LLMs (Nguyen et al., 2024), LLM-based QE does not perform well as of (SoTA) encoder-based models like CometKiwi (Rei et al., 2023).
The prediction of DA scores for QE constitutes regression task, requiring fine-grained numerical predictions. However, instructing LLMs to predict a DA score through prompt engineering instruction tuning for the generation task lacks regression-specifie training objectives. This limitation often results in outputs with reduced granularity, thereby constraining their effectiveness in tasks like QE that demand precise scalar predictions (Zerva et al., 2024). Furthermore, current LLM-based methods for cross-lingual tasks such as QE predominantly rely on representations from only the final Transformer layer, overlooking potentially valuable intermediate layer embeddings. Recent findings by Kargaran et al. (2025) indicate that optimal cross-lingual alignment is not necessarily achieved at the final Transformer layer for low-resource languages.
To address these challenges, we introduce the Adaptive Layer Optimization for Translation Quality Estimation (ALOPE) framework, a novel approach that integrates dedicated regression heads and low-rank adapters (LoRA) within Transformer layers, facilitating efficient instruction-based fine-tuning with LLMs. Our framework systematically investigates Transformer layers with regression heads, enabling the identification of optimal configurations that maximize cross-lingual transfer learning for segment-level translation quality estimation. Beyond investigating layer-wise embeddings, ALOPE introduces two additional strategies dynamic weighting and multi-head regression, which leverage multiple Transformer layers for QF. Moreover, the framework is developed to be model- agnostic, enabling seamless integration of any pre-trained LLMs with minimal configuration, thus promoting ease of deployment and scalability.
We evaluate ALOPE by comparing its performance against zero-shot inference and standard instruction fine-tuned (SIFT) LLMs across four open-source models (<8B parameters) on eight low-resource language pairs. The results highlight that ALOPE consistently outperforms SIFT, demonstrating its effectiveness by identifying optimal Transforner layers with better cross-lingual transfer learning, which benefits the quality estimation. The main contributions of this paper are,
We propose ALOPE, an adaptable framework that integrates regression heads with LoRA, enabling efficient QE with any LLMs by facilitating optimal cross-lingual representations.
ALOPE consistently outperforms the best results achieved by standard instruction fine-tuned L.LMs across all eight low-resource language pairs evaluated.
ALOPE introduces two additional approaches, dyynmmi-weighting and multi-layer regression, which utilize combined Transformer layers strategies and outperform the LLMs baseline under most conditions evaluated.
2 Background
In recent years, significant advancements have been made in the field of quality estimation. The development of QE models has evolved from early feature engineering-basee based
2

---

Published as a conference paper at COLM 2025
approaches (Specia et al., 2013; 2015) to more sophisticated methodologies, progressing through basic neural networks (Kim et al., 2017) and deep neural network architectures (Ive et al., 2018; Kepler et al., 2019). Currently, Transformer-based architectures represent the state-of-the--ar in QE (Blain et al., 2023; Rei et al., 2022; Ranasinghe et al., 2020; Perrella et al., 2022; Moura et al., 2020; Baek et al., 2020).
Recent advancements have positioned LLMs as a powerful tool across various NLP tasks (Zhao et al., 2023). In the context of translation evaluation, Kocmi & Federmann (2023) introduced the GEMBA metric, using zero-shot prompting across nine GPT variants and four prompt types for three language pairs. While GEMBA achieved best results with reference translations, its performance in referenceless QE remained below SOTA, highlighting the continued challenges LLMs face in reference-free evaluation settings. Mujadia et al. (2023) investigated QE for low-resuurce languages by pre-tuning adapters on large English- Indic parallel corpus for machine translation and subsequently fine-tuning the model with supervised QE data. Their findings suggest that pre-tuning with machine translation data does not enhance QE performance for low-resource languages and that LLMs struggle to transfer MT knowledge effectively for QE tasks in these settings.
has been noted that LLM-based approaches continue to underperform compared to traditional predictor-estimator architectures in QE, particularly for sentence-level scoring (Zerva et al., 2024). Predictor-estimator models, which explicitly treat QE as regression task, provide finer-grained quality assessments by effectively distinguishing between varying translation qualities (Kim et al., 2017). In contrast, LLMs, which rely on prompt engineering or instruction tuning, tend to produce outputs with limited granularity, often defaulting to narrower scoring ranges. This lack of differentiation restricts their ability to model nuanced quality variations acToss translations, contributing to their comparatively weaker performance in QE (Kocmi & Federmann, 2023; Vu et al., 2024).
Cross-lingual transfer learning intended improve low-resuurce language performance by transferring knowledge from high-resource languages through shared linguistic features (Choenni et al., 2023; Pham et al., 2024). However, despite its promise, existing studies show that the approach has so far been more effective in high-resource settings (Zhu et al., 2024), while its benefits for low-resource languages remain limited due to data scarcity and linguistic diversity (Zhu et al., 2024). The work of Sindhujan et al. (2025) reinforces the pross-lingual challenge in QE with LLMs, highlighting the poor tokenization issue for low-resource languages. Nonetheless, optimizing cross-lingual transfer remains essential for improving LL.M-based QE.
Methodology
3.1 Dataset
Our study mainly focuses on the QE dataset, which is derived from the WMT QE shared task (Zerva et al., 2022; Blain et al., 2023). It contains source and translation pairs with human-annotated DA scores?, which measure the quality of the machine translations. The QE dataset spans eight low-resource language pairs: English to (Gujarati, Hindi, Marathi, Tamil, Telugu), and (Nepali, Estonian, Sinhala] to English. Hindi and Estonian, while considered mid-resource languages for machine translation (Nguyen et al., 2024), have limited resources available for quality estimation. We utilize the dataset as train and test splits, as detailed in Appendix A.
3.2 Experimental settings
We utilized the GEMBA prompt, which is simple and straightforward prompt proposed by Kocmi& & Federmann (2023) for estimating the quality of the machine-translated text as shown in Figure 1. All our experiments were conducted with open-source generative LLMs which have 8B parameters or less: LLaMA2-7B, LLaMA3.1-8BB LLaMA3.2-3B and Aya-
Mean DA score annotations for Indic-language pair test sets be shared upon request.
3

---

Published as a conference paper at COLM 2025
Score the follewing translation from (Source Language) to (Target Language) continuous scale from 100, where of means meaning preserved" and score of hundred means "perfect meaning grammar".
(Source Language) source: (Source Sentence)
(Target Language) translation: (Translated Sentence)
Score:
Figure 1: GEMBA prompt for quality estimation (Kocmi & Federmann, 2023)
expanse-8B Our experiments were conducted under three settings: zero-shot, standard instruction fine-tuning (SIFT) and ALOPE framework. In zero-shot experiments, we obtain the results directly from the pre-trained LLMs using vLLM framework (Kwon et al., 2023) and we have utilized the LLaMA-Pactory framework (Zheng et al., 2024) to perform SIFT.
All instruction-fin--tne finettnn experiments (SIFT and ALOPE) with QE data were conducted using integrated multilingual language-pair training, which involved combining training data from the eight low-resource language pairs mentioned in section 3.1. Inference was performed on specifc test sets to evaluate model performance.
3.3 ALOPE framework
LLMs are optimized for next-token prediction, excelling at generative tasks but struggling with regression-based goals like quality estimation (Sindhujan al., 2025). This limits their ability to capture fine-grained relationships between language features and numerical scores. Although LLMS refine context through multiple Transformer layers, standard practice usually predicts with the final layer to obtain the outputs.
Addressing these challenges, ALOPE enables LLMs to perform regression by integrating regression head within the low-rank adapted Transformer architecture. We analyze the impact of adaptive regression heads to determine which Transformer layers contribute most to enhancing the performance of LLMs for cross-lingua QE tasks with low-resource language pairs.
We employed LoRA (Low-Rank Adaptation) (Hu et al., 2022) for efficient fine-tuning, targeting the projection layers of the Transformer architecture. Rather than updating the introduce trainable low-rank matrices into the attention mechanism, allowing the model to instruction fine-tune for the QE task while keeping the original weights frozen as shown in Figure 2. The adaptation
full parameter set during fine-tuning,
weight matrix RRrxd and Rdxr are the trainable low-
W is represented as W = W + BA, where rank matrices and < d, where (the rank) is much smaller than the dimension d which ensures a significant reduction in the number of tunable parameters. In our experiments, the rank is set to 32. To further optimize memory usage and computational efficiency, we combine LoRA with 4-bit quantization (Dettmers et al., 2023). To enable regression- based prediction with LLMs, the ALOPE framework assigns a dedicated regression head integrated into the Transformer architecture. This component is designed to extract hidden representations from any selected LoRA-adapted Transformer layer, thereby facilitating quality estimation using layer-specific contextual embeddings, The hidden state extracted from the final token of the sequence from the chosen LoRA-adapted Transformer layer is passed through regression head as shown in Figure 2. Let hn be the hidden state of the final token:
h = H Rd
where k is the chosen Transformer layer, is the dimensionality of the hidden state, Hę represents the sequence of hidden states at layer and [-1] represents the index of the
aL.LaMa-2-7B, L- ILLAMA31--5R LLAM332-3B, Aya-expanse-SB
4

---

Published as a conference paper at COLM 2025
643
MSE loss
quarises wegda
BCR
aw AW-BA
(am)
Transformer loyer
Predicted ID Vector
Fort Porsord
Transformen layer
Actiotion Layer here
T0,030
Th
Transformer layer
Orgat)
Transformer layer
Por lianal
Encoding
Pruu
Head
Input Embedding
PROMPT (with Source Tranalation)
hru
Prus
Prou
Prcc
Prud
fruu
Pru
Regression Head
Regrassion Head
Heed
Regnession Heed
Hood
Hoad
Head
Regresdia
Arombieed
Zyconocyy nighted remblee enbedtage
Multi- Multi-head regression
Dynamic weighting
Figure 2: Presents the ALOPE framework. Regression heads can obtain the embeddings from any LoRA-adapted Transformer layers, enabling flexible adaptation to perform QE,
final token in the sequence Transformer layer k. The regression head consists of single linear layer that maps the high-dimensional hidden state h2 to the final scalar output. The linear activation function applied implicitly, as the linear layer performs an affine transformation:
g = Wr. JER
where W, E Raxx the weight matrix of the linear layer. The output ÿ is a single scalar value per sequence, representing the predicted quality score. To evaluate the model, the mean squared error (MSE) loss computed by measuring the average squared difference between the predicted values in the 1D vector and the actual target values.
The regression-specific adaptation with regression heads and LoRA-based fine-tuning can be considered independently. However, the focus of our proposed method is efficiency in scenarios. We specifically chose regression heads
computationally resource-constraine incorporated with LoRA for its parameter efficiency, reducing the computational overhead compared to full fine-tuning, and allowing modularity given a pre-hosted LLM. ALOPE is able to perform QE with LoRA-less regression heads too. A publicly available library was adopted for our implementation
In the ALOPE framework, the regression heads are assigned numerical indices, where negative values indicate Transformer layers indexed in reverse from the final layer. For example, as shown in the Figure 2, placing a regression head at layer- -1 (TL-1) corresponds to extracting the hidden state from the final Transformer layer, while layer-4 (TL-4) refers to the layer located four positions before the last. This indexing approach provides consistent reference for selecting lavers across different models with different numbers of Transformer layers. To identify the optimal Transformer layer ALOPE systematically

---

Published as a conference paper at COLM 2025
explores six candidate Transformer layers across all evaluated LLMs: the final layer (-1), intermediate layers (-7 and -11- a mid-range layer (-16), and lower-level layers (-20 and -24).
In addition to the layer-specific regression head approach (vanilla ALOPE), extend the ALOPE framework with two additional strategies- dyygiiie-dyamiiee weighting and multi-head regression, both of which leverage multiple Transformer layers to enhance translation quality estimation through multiple layer-level representation modelling.
3.3.1 Dynamic weighting
In the dynamic weighting approach, we extract sequence-level contextual representations from selected Transformer layers using auxiliary heads. Auxiliary heads are solely utilized to extract embeddings from specific Transformer layers and are not subjected to direct supervised training. To obtain sequence-level embedding from the Transformer layer we select the final token's 's hidden state ltg. Instead of treating these selected layers equally, we assign a trainable scalar weight wy each layer k, and normalize these weights using the softmax function to identify each layer's contribution:
exp(1kk
ag
for 1,...,L
Dj=1 exp(w;)
These weights are initialized uniformly and subsequently optimized during training via backpropagation. The final embedding is weighted sum of the individual layer embeddings:
Σ aghx
combinedd
This representation is passed through a regression head to produce the predicted translation quality score. The model is optimized by minimizing the mean squared error between the predicted score and the ground truth. We refer to this method dynamic weighting because it allows the model to dynamically learn to prioritize information from the most informative layers for the task of translation quality estimation.
3.3.2 Multi-head regression
In this approach, multiple regression heads are integrated into different Transformer layers, each tasked with independently approximating translation quality. To ensure a balanced optimization, a weighted aggregation of individual regression head losses employed before back-propagation. The computed final loss is back-propagated through the model and updates the selected heads. This approach enables simultaneous learning across multiple layers. By optimizing a combined loss across selected layers, the framework effectively integrates multi-layered contextual information. During inference, the final translation quality prediction is derived by averaging the aggregated outputs from these regression heads.
3.4 Evaluation & Metrics
We use Spearman's 's correlation (Sedgwick, 2014) between the DA mean (averaged human- annotated DA scores from three or more annotators) and predictions as our evaluation metric. Additionally,
performed the Williams significance test (Graham & Baldwin, 2014; Williams, 1959) to assess whether the top-performing model for each language pair achieved significantly higher Spearman correlation than other models.
Results and Discussion
Table 1 reports the Spearman correlation scores obtained under zero-shot evaluation, alongside the results from the ALOPE framework with regression heads placed at various Transformer layers (See section 3.3) ). The table also highlights cases where ALOPE yields improvements over standard instruction fine-tuning (SIFT) results with LLMss Notably, the

---

Published as a conference paper at COLM 2025
Model
En-Gu
En-Hi
En-Mr
En-Ta
En-Te
Et-En
Ne-En
Si-En
Apg
Zero-shot
LLaMA2-7B
0.006
-0.002
0.053
0.067
-0.016
0.168
0.153
0.144
0.072
LLaMA3.1-88
0.164
0.194
0.245
0.220
0.132
0.503
0.262
0.267
0.248
LLLMA332-3B
0.095
0.095
0.116
0.128
0.098
0.359
0.216
0.120
0.153
Aya-expanse-8B
0.123
0.135
0.089
0.117
0.049
0.315
0.352
0.330
0.189
Aug
0.097
0.106
0.126
0.133
0.066
0.336
0.246
0.275
LLaMA2-7B
0.563
0.4141
*06609
0,525 1
0,356 1
10.742
0,596
"0.565
0.546
TL
LLMMA3.1-8B
'0,594
0.469 1
'0.620
0,567
0,363
0.734
0.647
0.547
0.567
LLMMA3.2-38
'0.601
'0.477 t
0.636 f
0.580 f
0.348
0.735 f
"0.674 1
0.543 1
0.575
Aya-expanse-SB
0.068
0.178
0,219
-0.006
0.275
0.115
0.012
0.077
0.117
Aug
0.457
0.385
0.521
0.416
0.335
0.581
0.482
0.433
LLaMA2-7B
0.567
0.336
0.542
0.484
0.317
*07739
0.606
0.3731
0.520
TL
LLMMA3.1-8B
'0,590
0.477
*0.625
0.528
0.388
'0.744
0.638
0.544
0.567
1JMMA32--3B
0.606
0.479t
"0.617"
0.5851
"0.369t
0.751
0.6641
"0.553t
0.578
Aya-expanse-88
0.538
0.447
0.597
0.528
0.347
10.741
0.646
0.544 f
0.549
Aug
0.575
0.435*
0,595*
0.531
0.355
0.744*
0.639*
0.554
LLaMA2-7B
0.360
0.301
0,361
0,254
0,2931
0.405
0,164
0.049
0.273
TL
LLaMA3.1-8B
0.514
0.4121
"0.609"
0.438
0,3041
0.148
0,554
0.493
0.434
LLMMA32-38
'0,594
'0.4761
0.6051
0.6101
'0,3731
'0.7481
'066781
"0.5601
0,581
Aya-expanse-88
0.490
0.4111
0.5721
0.445
0.3361
0.569
0.453
0.4397
0.464
Aug
0.489
0.400
0.537
0.437
0,327
0.467
0,462
0.385
LLaMA2-7B
0.540
0,381
0.585 f
0.482 f
0.308
107751
0.580
'0.569
0.524
T1
LLaMA3.1-8B
0,558
0.453
0.602 1
0.523
0,350
10,737
0.652
0.513
0.548
LLaMA32-3BB
0,557
"0.459 f
0.597 1
0.547
0.338
00.74
0.6821
"0.567 t
0.561
Aya-expanse-8B
0.467
0.390
0.357 f
0.481
0.314
0.727
0.576
0.540 f
0.506
Aug
0.530
0.421
0.585
0.508
0.327
0.740
0.622
0.547
LLAMA2-7B
0.470*
0.405t
0.5441
0,4601
0.3381
0.6841
0.5081
0.534
0.493
TL (220
LLaMA3.1-8B
0.484
0,394
0.5531
0,321
0.172
0.649
0.524
0.494
0.449
LLMMA32-38
0.430
0.408T
0.5791
0.303
0,2861
0.601
0,488
0.4641
0.445
Aya-expanse-8R
0.437
0.300
0.488
0.263
0.287
0.483
0.438
0.3951
0.386
Aug
0.455
0.377
0.541
0.337
0.271
0.604
0.490
0.472
LLaMA2-7B
0.500
0.4211
0.5381
0.379
0.239
0.630t
0.5071
U.472
0.461
(-2)
LLaMA3.1-BB
0.421
0.3781
0.5521
0.330
0,2901
0.515
0.530
0.461
0.435
L-aMA322-3
0.443
0.376
0,507
0,367
0,299
0.559
0.528
0.487
0.446
Aya-expanse-8B
0.375
0.319
0.440
0.337
0.220
0.393
0.407
0.345
0.354
Aug
0.435
0.373
0.509
0.353
0.262
0.524
0.493
0.442
Table 1: Spearman correlation scores for zero-shot inference and ALOPE experiments across different Transformer layers (TL). TL(11) refers the final Transformer layer, TL(-7), TL(-11) are intermediate layers, TI-(166 is mid-layer, and (T2(-,), (T-(-2) are lower-level layers. 1 indicates cases where ALOPE outperforms standard instruction fine-tuning (SIFT) for given language pair (Correlation scores for SIFT can be found in Appendix B). Bolded values denote the highest correlation scores per language pair; (') marks the highest average across Transformer layers; (') indicates the statistically insignificant Spearman scores compared to the best scores. The dashed line separates language pairs where English is the target language.
performance under zero-shot settings is substantially lower across all eight low-resource language pairs when compared both SIFT and ALÖPE. When benchmarked against the best Spearman scores from SIFT (refer to Appendix B), ALOPE obtains the best correlation scores for all evaluated language pairs.
4.1 Layer-specific adaptation for QE
The experimental results obtained using the ALOPE framework, presented in Table 1, reveal that the most effective performance across language pairs is not achieved when the regression head is placed at the final Transformer layer (TL-1), but rather when it is positioned at intermediate layers (TL-7, TL-11) between the final and mid-level layers.
7

---

Published as a conference paper at COLM 2025
Among all layers evaluated, TL-7 consistently yielded the highest Spearman correlation scores for the majority of language pairs, demonstrating both top-performing outcomes and a high number of instances with statistically insignificant differences from the best scores. The layer-wise averages shown in Table 1 also confirm that TI--7 consistently offers the highest average correlation across models and language pairs. These suggest that TL-7 provides robust and reliable contextual representations suitable for quality estimation of low-resource language pairs.
Additionally, TL-11, while producing the highest score for En-Ta, exhibited statistically insignificant differences from the best-performing layer across the other seven language pairs. This further reinforces the effectiveness of intermediate layers in capturing cross- lingual alignment. Collectively, these results indicate that optimal layer selection is key factor in enhancing QE performance with LLMs and that intermediate Transformer layers are better suited for extracting representations that support accurate quality prediction in low-resource machine translation tasks.
At the mid-layer (TL-16), the Ne-En language pair achieved the highest correlation, while three other language pairs (En-Hi, Et-En, and Si-En) exhibited performance that was statistically indistinguishable from the best scores, as shown in Table 1. Notably, three out of these four language pairs share common characteristic: English serves as the target language in the translation direction. This pattern suggests that cross-lingual transfer learning in Transformer-ban bbssed LLMs reaches representational consistency earlier from the mid-layer onwards, when English is the target language. The earlier stabilization may be attributed to the extensive exposure of LLMs to English during pre-training, which enhances cross-lingual alignment when generating English translations (Kargaran et al., 2024). Furthermore, as shown in Table 1, models consistently achieve higher average correlation scores when English is the target language in MT, indicating stronger cross- lingual transfer in that direction. Notably, Et-En yields the highest layer-wise average across all experimental settings, likely due to the shared Latin-based script, reinforcing prior findings on the role of script similarity in enhancing cross- lingual transfer (Sindhujan et al., 2025).
The overall results from ALOPE indicate a decrease in correlation scores beyond the mid-Transforme layer (TL-16) for most models and language pairs. This aligns with the observation of Kargaran et al. (2024), that as embeddings for low-resource languages when progressed through the mranTraarformer layers, they became more aligned with the primary language of the LLMss. This improved alignment likely contributes to better cross-lingual representation at intermediate layers (TL-11, TI.-7) after the mid-layers, and may also explain the decreased performance observed in the lower-level Transformer layers (TL-20, and TLL24), where such alignment has not yet stabilized.
4.1.1 Model-wise analysis
The pest-perforring models under the ALOPE framework consistently belonged to the LLaMA family. Notably, the LLaMMA 3.2 model yielded the highest Spearman correlation scores for six out of eight language pairs: En to (Gu, Hi, Mr, Ta} and ( (Et, Ne) to En. LLaMA 3.1 demonstrated the best performance for En-Te, while LLaMA 2-7B outperformed other models on Si-En. These results highlight the effectiveness of LLaMA-based models for cross-lingual tasks over other open-source models.
Furthermore, statistical analysis using the Williams significance test revealed that although LLaMA 3.2 did not yield the absolute highest correlation scores for En-Te or Si-En, its performance at TL-7 and TL-11 was statistically indistinguishable from the best-performing models (as detailed in Table 1). Notably, LLaMA 3.2 attained this level of performance despite having the smallest parameter size (3 billion) among all the models evaluated in this study. This suggests that model size alone is not the sole determinant of the performance of a cross-lingual task such as QE and highlights the efficiency and adaptability of LLaMA 3.2 within resource-constrained environments. The Aya model also demonstrated correlation scores comparable to LLaMA models, with one notable exception, when the ALOPE experimented with the final Transformer layer (TL-1), its performance exhibited significant decrease. The possible reason for this could be that the final layer of Aya is
8

---

Published as a conference paper at COLM 2025
Multihead regression
Standard instruction fine-tuning (SIFT)
Dynamic weighting
vanilla ALOPE
0.7
Correaation Score
Correlation Score
Correlation Score
0.8
0.5
0.4
0.3
0.2
En-lee
Et-En
Ne-En
S--n
Conelation Score
Conrelation Score
Conrelation Score
0.6-
Comelation
0.4
0.2
Figure LLLMA332-3B results for dynamic-weighting and multi-head regression compared against standard instruction fine-tuning results and best of Vanilla ALOPE. Y-axis represents the correlation score and the X-axis represents the selected range of Transformer layers for dynamic weighting and multi-head regression experiments.
less aligned with quality estimation signals, whereas its intermediate representations still capture useful features. Overall, all the LLMs demonstrated improved performance when using ALOPE compared to SIFT, when the regression heads were attached to the optimal Transformer layers.
Dynamic weighting & Multi-head regression for QE
To further evaluate the benefits of leveraging multiple Transformer layers, dynamic weighting and multi-head regression experiments were conducted using the LLaMA 3.2-
3B model, which had previously demonstrated the most consistent performance across language pairs (Section 4.1). The experiments focused on three Transformer layer groupings: TL-l to TL-7, TL-8 to TL-11, and TL-12 TL-16. As illustrated in Figure 3, both approaches yielded improvements over SIFT for En-(Gu, Mr, Te} and Si-En. For the En-Hi language pair, the multi-head regression approach outperformed SIFT across all three layer groupings, whereas dynamic weighting exhibited a slight performance drop in the TL-8 to TT.-11 configuration. In the case of Et-En and Ne-En, both multi-layer strategies delivered comparable performance to SIFT, with only marginal deviations observed. However, the En-Ta pair continued to yield suboptimal results under both approaches, resembling its weaker performance trends observed in previous experiments. This underscores the unique challenges associated with En-Ta in cross-lingual QE tasks.
In Figure 3, Vanilla ALOPE represents the best correlation scores obtained through layer- specific adaptation with LLaMA3.2-3B model, as detailed in Section 4.1. When comparing the performance of multi-layer approaches with Vanilla ALOPE's layer-specific ada adaptation, Vanilla ALOPE generally achieves higher correlation scores. Nevertheless, both multi-layer approaches demonstrate improvements over SIFT for the majority of the language pairs. These results highlight that multi-layer methods can also effectively leverage complementary information from multiple Transformer layers, making them valuable alternatives for efficient LLM-based translation quality estimation.
4.3 Comparative analysis: ALOPE us. QE frameworks Monolingual setting
The ALOPE approach demonstrated consistent improvements in correlation scores over SIFT across all language pairs and achieved results comparable to those of established

---

Published as a conference paper at COLM 2025
Method
En-Gu
En-Hi
En-Mr
En-Ta
En-Te
Et-En
Ne-En
Si-En
ALOPE"
0.606
0.479
0.636
0.610
0.388
0.751
0.682
0.573
SIFT-IIMMM* *
0.555
0.393
0.530
0.586
0.290
0.728
0.618
0.558
TransQuest 0.630
0.478
0.606
0.603
0.358
0.760
0.718
0.579
CometKiwi 0.637
0.546
0.635
0.616
0.338
0.860
0.789
0.703
Table 2: Comparison highest Spearman correlation scores across ALOPE, standard instruction fine- and CometKiwi. Methods marked with ()) are trained using the datasets described Appendix A. The CometKiwi results are from the publicly available model pre-trained on over 940K QE data samples (Rei et al., 2023).
tuning (SIFT) with LLMs, and encoder-based QE Maaelines-TTranSCC
pre-trained encoder-based QE frameworks, shown in Table 2. Notably, for the En- Mr and En-Te language pairs, ALOPE even surpassed the SOTA performance of these frameworks. While pre-trained encoder-based models such as TransQuest (InfoXLM-LLrgee) and CometKiwi (XLM-R-XI.) continue to show strong performance, they require dedicated training pipelines and significant amount of data for fine-tuning, In contrast, ALOPE offers a flexible and modular solution that can be applied to pre-deployed LLMs without extensive reconfiguration, making it practical solution for performing QE.
Despite being based on LLMs, ALOPE demonstrates favourable memory efficiency when compared to encoder-based QE frameworks. Although LLMs are often considered more resource-intensive, our implementation of ALOPE- ALOPE-augmented with LoRA-adapted lightweight regression heads exhibits competitive GPU memory usage, as detailed in Appendix C. This indicates that ALOPE can be deployed with overheads comparable to, or even lower than, those of established encoder-basee models, highlighting its practicality in snouucceconsstrained environments. Finally, it is also scalable to unified evaluation and correction framework where an additional causal head can provide translation error reasoning-based corrections perform automatic post-editing.
To evaluate the generalizability of ALOPE for regression tasks beyond cross-lingual quality estimation, conducted ablation study, which is detailed in the Appendix E, observing its performance on a monolingual regression task. Results demonstrated that ALOPE consistently outperforms standard instruction fine-tuning and baseline scores in the monolingual setting, achieving the highest scores across evaluated languages- English, Spanish and Arabic. These findings highlight ALOPE's ability to effectively identify optimal transformer layers for enhanced performance in both monolingual and rrossllnngual settings.
5 Conclusion
This paper introduced ALOPE, a novel framework for improving segment-level translation quality estimation using LLMs. By integrating regression heads with LoRA and strategically leveraging intermediate Transformer layers, ALOPE outperforms standard instruction fine-tuning results of LLMs and achieves results par with, or better than, encoder- based QE frameworks. Our experiments demonstrate that intermediate Transformer layers, particularly TL-7, TL-7, yield more effective cross-lingual representations for QE, while performance tends to decrease in layers beyond the mid-range. Notably, LLaMA 3.2 emerged as the most effective model across language pairs, despite having the smallest parameter size, underscoring that parameter count is not the sole indicator of QE effectiveness. Furthermore, early stabilization of the performance was observed when English served as the target language in translation, especially around mid-layer depths. Additionally, the framework introduces dynamic weighting and multi-head regression approaches, which leverage selective set of multiple Transformer layers to perform QE. ALOPE demonstrates competitive GPU efficiency, reinforcing its practicality as a flexible and scalable solution for deploying LLMs in resource-cosstrined settings to perform QE. We also envision this framework to be extended for reasoning over MT errors informed by regression-head predicted DA, providing more reliability to error reasoning and MT assessment.
10

---

Published as a conference paper at COLM 2025
References
Yujin Back, Zae Myung Kim, Jihyung Moon, Hyunjoong Kim, and Eunjeong Park. Patquest: Papago translation quality estimation. In Proceedings of the Fifth Conference on Machine Translation, pp. 991-998, 2020.
Frederic Blain, Chrysoula Zerva, Ricardo Rei, Nuno M. Guerreiro, Diptesh Kanojia, José G. C. de Souza, Beatriz Silva, Tânia Vaz, Yan Jingxuan, Fatemeh Azadi, Constantin Orasan, and André Martins. Findings of the WMT 2023 shared task on quality estimation. In Philipp Koehn, Barry Haddow, Tom Kocmi, and Christof Monz (eds.), Proceedings of the Eighth Conference on Machine Translation, pp. 629-653, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/vl 2tt//2223..mt.....
URL
https://aclanthology/ org/ wmt-1.52.
Rochelle Choenni, Dan Garrette, and Ekaterina Shutova. Cross-lingual transfer with language-specificc subnetworks for low-resource dependency parsing. Computational Linguistics, PP. 613-641, September 2023. doi: 10.1162/coli a 00482. //aclanthology. Secey.org/220 www.:org.......oooo cl-3.3
URL https:
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized lims, 2023.
Yvette Graham and Timothy Baldwin. Testing for significance of increased correlation with human judgment. In Alessandro Moschitti, Bo Pang, and Walter Daelemans (eds.), Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 172-176, Doha, Qatar, October 2014. Association for Computational Linguistics. doi: 10.3115/11DD4-1022 URL httpp:://aclaathooooo.ooo/////// org/D14-1020.
Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. Continuous measurement scales in human evaluation of machine translation. In Antonio Pareja-Lora, Maria Liakata, and Stefanie Dipper (eds.), Proceedings of the 7th Linguistic Annotation Workshop and Interoperabilityy with Discourse, pp. 33 33-41, Sofia, Bulgaria, August 2013. Association for Computational Linguistics. URL https: haclat///aclattholooyyy org/W13-2300
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https: http:///ppeeeeee.eeeeee net/ forum?iidnnZeWeeeFvyy
Julia Ive, Frédéric Blain, and Lucia Specia. Deepquest: a framework for neural-based quality estimation, In Proceedings of the 27th International Conference Computational Linguistics, pp. 3146-3157, 2018.
Amir Hossein Kargaran, Ali Modarressi, Nafiseh Nikeghbal, Jana Diesner, François Yvon, and Hinrich Schütze. Mexa: Multilingual evaluation of english-centric 1lms via cross- 05873.
lingual alignment, 2024. URL
Amir Hossein Kargaran, Ali Modarressi, Nafiseh Nikeghbal, Jana Diesner, François Yvon, and Hinrich Schuetze. MEXA: Multilingual evaluation of English-centric LLMs via cross-lingual alignment. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Findings of the Association for Computational Linguistics: ACL 2025, pp. 27001-27023, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-256-5, URL https: http:///aaaaantoogo.oooooo .rgg/ 2025. findings-ace 1385/.
Fabio Kepler, Jonay Trénous, Marcos Treviso, Miguel Vera, and André FT Martins. Openkiwi: An open source framework for quality estimation. arXiv preprint ar arXiv:1902.0866 2019.
Hyun Kim, Hun-Young Jung, Hongseok Kwon, Jong-Hyeok Lee, and Seung-Hoon Na. Predictor-estimator: neural quality estimation based target word prediction for machine translation. ACM Transactions on Asian and Processing (TALLIP), 17(1):1-22, 2017.
Language Information
11

---

Published as a conference paper at COLM 2025
Tom Kocmi and Christian Federmann. Large language models are state-of-the-arr evaluators of translation quality. In Mary Nurminen, Judith Brenner, Maarit Koponen, Sirkku Latomaa, Mikhail Mikhailov, Frederike Schierl, Tharindu Ranasinghe, Eva Vanmassenhove, Sergi Alvarez. Vidal, Nora Aranberri, Mara Nunziatini, Carla Parra Escartín, Mikel Forcada, Maja Popovic, Carolina Scarton, and Helena Moniz (eds.), Proceedings of the 24th Annual Conference of the European Association for Machine Translation, pp. 193-203, lampere, Finland, June 2023. European Association for Machine Translation. URL https::/aallantholoog.. org/ 2223.2023.22 eamt-1.19 19.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Arle Richard Lommel, Aljoscha Burchardt, and Hans Uszkoreit. Multidimensional quality metrics: flexible system for assessing translation quality. In Proceedings of Translating and the Computer 35, London, UK, November 28-29 2013. Aslib. URL org/2013. 3. 1c-1. 6/.
Saif Mohammad, Felipe Bravo-Marquez, Mohammad Salameh, and Svetlana Kiritchenko. SemEval-2018 task 1: Affect in tweets. In Marianna Apidianaki, Saif M. Mohammad, Jonathan May, Ekaterina Shutova, Steven Bethard, and Marine Carpuat (eds.), Proceedings of the 12th International Workshop on Semantic Evaluation, pp. 1-17, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/v1/S18- https://aclanthon
URL
org/S1 2/////18-10111111 1001.
Joao Moura, Miguel Vera, Daan van Stigt, Fabio Kepler, and André FT Martins. Tst-unbabel participation in the wmt20 quality estimation shared task. In Proceedings of the Fifth Conference on Machine Translation, pp. 1029-1036, 2020.
Vandan Mujadia, Pruthwik Mishra, Arafat Ahsan, and Dipti M. Sharma. Towards large language model driven reference-less translation evaluation for English and Indian language. In Jyoti D. Pawar and Sobha Lalitha Devi (eds.), Proceedings of the 20th International Conference on Natural Language Processing (ICON), pp. 357-369, Goa University, Goa, India, December 2023. NLP Association of India (NLPAI). URL https://aclantholl.. hww.aanntoooooooooo/////22 org/2023. 1...
Xuan-Phi Nguyen, Mahani Aljunied, Shafiq Joty, and Lidong Bing. Democratizing LLMs for low-resource languages by leveraging their Englishh dominant abilities with linguistically- diverse prompts. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3501-3516, Bangkok, Thailand, August 2024. Association for Computational Linguistics. URL https: hatpaaaaannntoooooyoooooo wrg./024........... acl long. 192.
by
Stefano Perrella, Lorenzo Proietti, Alessandro Scirè, Niccolò Campolungo, and Roberto Navigli. Matese: Machine translation evaluation as a sequence tagging problem. In Proceedings of the Seventh Conference on Machine Translation (WMT), pp. 569-577, 2022.
Trinh Pham, Khoi Le, and Anh Tuan Luu. UniBridge: A unified approach to cross-lingual transfer learning for low-resource languages. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Pp. 3168-3184, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653 222.2222..allloogg 174. URL https://aclanthol..o wwwp:/looggo.g//22200 wrg. acl- 1ong. 174.
Shenbin Qian, Archchana Sindhujan, Minnie Kabra, Diptesh Kanojia, Constantin Orasan, Tharindu Ranasinghe, and Fred Blain. What do large language models need for machine translation evaluation? In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Pp. 3660-3674, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v /1t/://11222...mmmmmaaaaa 214. URL https: /aclaantoooooyyyyyoo org/ 2024. emnlp-main.211 214/.
12

---

Published as a conference paper at COLM 2025
Tharindu Ranasinghe, Constantin Orasan, and Ruslan Mitkov. TransQuest: Translation quality estimation with cross-lingual transformers. In Donia Scott, Nuria Bel, and Chengqing Zong (eds.), Proceedings of the 28th International Conference on Computational Linguistics, pp. 5070-5081, Barcelona, Spain (Online), December 2020. International Committee on Computational Linguistics. doi: 10.18653/vi v1/66///12220.coliggg 72000.coling-mainn4555 URL https://aclantthologg org/2020 malinngmmin..ee 445/.
Ricardo Rei, Marcos Treviso, Nuno M. Guerreiro, Chrysoula Zerva, Ana C Farinha, Christine Maroti, José G. C. de Souza, Taisiya Glushkova, Duarte Alves, Luisa Coheur, Alon Lavie, and André F. T. Martins. CometKiwi: IST-unbabel 2022 submission for the quality estimation shared task. In Philiip Koehn, Loic Barrault, Ondřej Bojar, Fethi Bougares, Rajen Chatterjee, Marta R. Costa-jusså, Christian Federmann, Mark Fishel, Alexander Fraser, Markus Freitag, Yvette Graham, Roman Grundkiewicz, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Tom Kocmi, André Martins, Makoto Morishita, Christof Monz, Masaaki Nagata, Toshiaki Nakazawa, Matteo Negri, Aurélie Névéol, Mariana Neves, Martin Popel, Marco Turchi, and Marcos Zampieri (eds.), Proceedings of the Seventh Conference on Machine Translation (WMT), pp. 634-645, Abu Dhabi, United Arab Emirates (Hybrid), December 2022. Association for Computational Linguistics. URL https::: http::::/aaaaaaaatah/ http///aaaaantooggooyooooo wtwphhoooog.oooooooooooooo org.oogg///// wmt-1. 1.60.
Ricardo Rei, Nuno M. Guerreiro, JosÅ@ Pombal, Daan van Stigt, Marcos Treviso, Luisa Coheur, José G. C. de Souza, and André Martins. Scaling up CometKiwi: Unbabel-IST 2023 submission for the quality estimation shared task. In Philipp Koehn, Barry Haddow, Tom Koemi, and Christof Monz (eds.), Proceedings of the Eighth Conference on Machine Translation, pp. 841-848, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v
URL https: ac aclanthology. org/ 2023. wmt-1 73/.
Philip Sedgwick. Spearman's rank correlation coefficient. Bmj, 349, 2014.
Archchana Sindhujan, Diptesh Kanojia, and Constantin Orasan. Optimizing quality estimation for low-resuurce language translations: Exploring the role of language relatedness. In Proceedings of the Conference New Trends in Translation and Technology 2024. Incoma, 2024.
Archchana Sindhujan, Diptesh Kanojia, Constantin Orasan, and Shenbin Qian. When LLMs struggle: Reference-less translation evaluation for low-resource languages. In Hansi Hettiarachchi, Tharindu Ranasinghe, Paul Rayson, Ruslan Mitkov, Mohamed Gaber, Damith Premasiri, Fiona Anting Tan, and Lasitha Uyangodage (eds.), Proceedings of the First Workshop on Language Models for Low-Resource Languages, pp. 437-459, Abu Dhabi, United Arab Emirates, January 2025. Association for Computational Linguistics. URL loreslm-1 1.33/.
https://aclanthholgg..oogg
Lucia Specia, Kashif Shah, José GC De Souza, and Trevor Cohn. Quest-a translation quality estimation framework. In Proceedings of the 51st Annual Meeting of the Association for Compulational Linguistics: System Demonstrations, pp. 79-84, 2013.
Lucia Specia, Gustavo Henrique Paetzold, and Carolina Scarton. Multi-level Translation Quality Prediction with QUEST++. pp. 115-120. Association for Computational Linguistics and The Asian Federation of Natural Language Processing, 2015. doi: (0.3115/v1/P15-400 URL
org.o/5-4/2
Tu Vu, Kalpesh Krishna, Salaheddin Alzubi, Chris Tar, Manaal Faruqui, and Yun-Hsuan Sung. Foundational autoraters: Taming large language models for better automatic evaluation. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Prucessing, pp. 17086-17105 Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/vo http:/2222emmmpmmaaa 949/.
URL https: htt.t:ttss//allee http::::///allaahhooooooooo org/ emnlp-man main.
E.J. Williams. Regression Analysis. WILEY SERIES in PROBABILITY and STATISTICS: APPLIED PROBABILITY and STATIST ICS SECTION Series. Wiley, 1959. ISBN 9780471946779. URL https://bool google. http/bookksiiixxxxxAAA...
13

---

Published as a conference paper at COLM 2025
Chrysoula Zerva, Frédéric Blain, Ricardo Rei, Piyawat Lertvitayyakumjom, José G. C. de Souza, Steffen Eger, Diptesh Kanojia, Duarte Alves, Constantin Oräsan, Marina Fomicheva, André F. T. Martins, and Lucia Specia. Findings of the WMT 2022 shared task on quality estimation. In Philipp Koehn, Loic Barrault, Ondfej Bojar, Fethi Bougares, Rajen Chatterjee, Marta R. Costa-jussä, Christian Federmann, Mark Fishel, Alexander Fraser, Markus Freitag, Yvette Graham, Roman Grundkiewicz, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Tom Kocmi, André Martins, Makoto Morishita, Christof Monz, Masaaki Nagata, Toshiaki Nakazawa, Matteo Negri, Aurélie Névéol, Mariana Neves, Martin Popel, Marco Turchi, and Marcos Zampieri (eds.), Proceedings of the Seventh Conference on Machine Translation (WMT), pp. 69-99, Abu Dhabi, United Arab Emirates (Hybrid), December 2022. Association for Computational Linguistics. URL wwwp:/hhooooooooo///g2/////// wmt-1.3.
Chrysoula Zerva, Frederic Blain, José G. C. De Souza, Diptesh Kanojia, Sourabh Deoghare, Nuno M. Guerreiro, Giuseppe Attanasio, Ricardo Rei, Constantin Orasan, Matteo Negri, Marco Turchi, Rajen Chatterjee, Pushpak Bhattacharyya, Markus Freitag, and André Martins. Findings of the quality estimation shared task at WMT 2024: Are LLMs closing the gap in QE? In Barry Haddow, Tom Kocmi, Philipp Koehn, and Christof Monz (eds.), Proceedings of the Ninth Conference on Machine Translation, pp. 82-109, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. 10.18653/v1 2003//v1122/// wmt-1.3. URL heatpt://aclanthoolggyy org/ 2024. wmt-1.33 3/.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jin Nie, and Ji-Rong Wen. A survey of large language models, 2023.
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. Xiv preprint arXie:2403.1337,, 2024. URL http:://rxxiv wwwpiii..rg//bbse// org/abs/2403 13372.
Shaolin Zhu, Supryadi, Shaoyang Xu, Haoran Sun, Leiyu Pan, Menglong Cui, Jiangcun Du, Renren Jin, Ántónio Branco, and Deyi Xiong. Multilinuual large language models: A systematic survey, 2024. URL https::/arxiee wrgg/bs/2411111.1.. http:://arrdd..orr/////11111111 11072.
14

---

Published as a conference paper at COLM 2025
A
Dataset
En-Gu
En-Hi
En-Mr
En-Ta
En-Te
Et-En
Ne-En
Si-En
Train
7000
7000
26000
7000
7000
7000
7000
7000
Test
1000
1000
699
1000
1000
1000
1000
1000
Table 3: Dataset splits of QE dataset utilized in our study, Experiments were conducted with
8 low-resource language pairs to evaluate model performance.
B Results of standard instruction fine-tuned LLMs (SIFT)
Models
En-Gu
En-Hi
En-Mr
En-Ta
En-Te
Et-En
Ne-En
Si-En
Aug
LLaMA2-7B
0.454
0.319
0.466
0.449
0.266
0.591
0.478
0.374
0.425
LLaMA3.1-88
0.555
0.368
0.530
0.586
0.233
0.728
0.618
0.558
0.522
LLaMA3.2-3B
0.501
0.393
0.524
0.567
0.224
0.678
0.587
0.451
0.491
Aya-expanse-8B
0.528
0.388
0.515
0.496
0.290
0.605
0.568
0.384
0.472
Aug
0.510
0.367
0.509
0.525
0.253
0.650
0.563
0.442
Table 4: Spearman correlation score obtained with standard instruction fine-tuning. Bolded values denote the highest Spearman score obtained for each language pair
C GPU memory utilization
Encoder
Memoryy Usage (GB)
TranQueest (Info [nfooLel-Large] Large)
Cometkiwi23 (XLMR-XL)
LLaMA2-18
LLaMA3.1-8
LLaMA3.2-38
Aya expanse-88
Figure 4: GPU memory utilization comparison between encoder-based QE models and various LLM-based ALOPE configurations.
15

---

Published as a conference paper at COLM 2025
D
Language pair-wise performance across different Transformer layers
LLaM42-78
LLaMA3.1-88
LLaMA3.2-38
En-Cu
En-li
06
0.93
Speamann Srat
Syertmn
223
01
11-33
Ri/ko
114-293
BL:24
RE-11
T26661
Bi-201
Tu:28
Settrna
Setting
En-Mr
En-Ta
06
Sparmen Souee
Spairmen
R60
12/41)
T(-46)
B8-43
RL-7
71-46
Settige
Setting
bn-le
81-8n
0.7
Spoa TONTTTI Scur
8.30
6.4
020
0.2
72-13
TL:20
SeWingg
Setting
ke-En
S4En
0.
Spaamer Sooee
Spaarmen
0.4
0.
R2-11
811311
22-200
16:10
RiC-8)
Setting
Figure 5: Spearman correlation scores obtained across different Transformer layers with ALOPE layer-specific regression experiments.
16

---

Published as a conference paper at COLM 2025
E Assessing the generalization of ALOPE across regression tasks: An ablation study
Uima 3.1-88
Uama 3.2-38
1.0
Crocs ingua
Cross Ingusl
Sinerman
0.4
0.4
611
T1 (-)
TI 6/00
TV 1-200
11 (10)
T3 6-)
1-00
T1 1-44
Ulama-2-77 2.7b
Aya-expanse-Be 4bb
1.0
Grosp Ingua Voralings
Cross ligusl Monsiingaa
0.4
0.4
Arg.
(1)
T1 (-))
T1 6100
TV 1-00
(24]
T1 (10)
71
T1.010
T1 1-20
T1 1.4
Figure 6: Model-wise average Spearman correlation scores obtained with layer-specific ALOPE approach, comparing performance on monolingual versus cross-lingual regression tasks.
The ALOPE framework consistently outperformed standard instruction fine-tuned LLMs in the cross-lingual regression task of quality estimation across all language pairs and delivered results comparable to leading encoder-based QE models (Table 2). For En-Mr and En-Te, it even exceeded state-of-the of-the-art performance. To better understand whether the performance gains observed with ALOPE are specific to cross-lingual QE or indicative of broader regression capabilities, conducted ablation study using monolingual regression task.
For the comparative analysis, utilized the SemEval-2018 shared task dataset (Mohammad et al., 2018), which focuses on predicting emotion intensity scores, real-valued outputs ranging from 0 to based tweet content. The dataset spans four emotion categories and comprises monolingual data in English, Spanish, and Arabic. SemEval was selected for its structural similarity to the QE dataset, as both tasks involve predicting continuous score based on the text input. However, the key distinction lies in language composition: SemEval monolingual, whereas QE contains target translation data, which is rross-lingual. This contrast makes SemEval suitable benchmark for evaluating ALOPE's performance in monolingual versus cross-lingual regression settings.
Experiments were conducted using the same set of LLMs within the ALOPE framework, demonstrating its flexibility in adapting monolingual regression tasks. As shown the Table 6, all three languages obtained the best score with ALOPE for emotion intensity score prediction showing improved correlation score over the baseline, i.e., best score obtained for the SemEval-2018 shared task (Mohammad et al., 2018) and standard instruction fine-tuned results of LLMs. This shows the effectiveness of ALOPE for LLM-based regression tasks.
Although direct comparison between monolingual and cross-lingual tasks is not feasible due to their differing objectives, we analyzed model-wise average Spearman correlation scores across selected Transformer layers (see Figure 6). This graph shows that the monolingual
17

---

Published as a conference paper at COLM 2025
You are provided with tweet and specific emotion. Your task is to determine the intensity that emotion expressed the tweet. The intensity score should a real value between (indicating the least amount of emotion) and 1 (indicating the most intense level of that emotion). Focus understanding the context, tone, and emotion of the tweet accurately determine the emotional intensity.
Tweet: (tweet)
Emotion: (emotion)
Provide intensity score between and 1 that best represents the emotional state of the person tweeting.
Figure 7: Prompt for emotion intensity prediction
English
Spanish
Arabic
Train
7102 4068
4544 2616
3376
Test
1563
Table 5: Data count of each split utilized for emotion intensity score prediction.
regression task consistently achieves higher average correlation scores across most of the Transformer layers, including the lower-level layers (TL:-20 and -24). contrast, cross- lingual QE tasks show improved performance predominantly in the intermediate layers beyond the mid-level. These findings reinforce ALOPE's effectiveness in identifying and leveraging the most suitable Transformer layers for cross-lingual quality estimation, thereby enhancing performance in low-resource settings.
Building on this analysis, the ALOPE framework demonstrates strong adaptability and consistently outperforms standard LLM baselines across both monolingual and tross-lingual regression tasks, highlighting its generalizability and effectiveness in identifying optimal transformer layers for varied language settings.
18

---

Published as a conference paper at COLM 2025
Model
English p
Spanish
Arabic
p
llama-2-7b
0.766
0.767
0.775
0.778
0.481
0,486
llama 3.1-8B
0.794
0.792
0.797
0.798
0.662
0.670
SIFT
Ilama 3.2-3B
0.777
0.775
0.755
0.752
0.546
0.558
aya-expanse-8b
0.779
0.782
0.807
0.807
0.721
0.727
Avg
0.779
0.783
0.602
llama-2-7b
0.755
0.759
0.321
0.320
0.123
0.116
Ilama 3.1-8B
0.786
0.788
0.709
0.709
0.471
0.480
Ilama 3.2-3B
0.773
0.771
0.706
0.705
0.438
0.440
TL
aya-expanse-88
0.767
0.771
0.760
0.760
0.510
0.516
Aog
0.770
0.624
0.385
llama-2-7b
0.747
0.748
0.410
0.407
0.239
0.238
Ilama 3.1-8B
0.823
0.825
0.812
0.815
0.665
0.670
Ilama 3.2-3B
0.801
0.803
0.787
0.791
0.553
0.556
aya-expanse-8b
0.686
0.683
0.657
0.662
0.482
0.490
Aeg
0.764
0.666
0.485
llama-2-7b
0,801
0.804
0.748
0.750
0.111
0.104
TL (-16)
Ilama 3.1-8B
0,830
0.829
0.840
0.825
0.707
0.712
Ilama 3.2-3B
0.802
0.804
0.791
0.794
0.611
0.611
aya-expanse-8b
0,826
0.830
0.839
0.841
0.769
0.779
Avg
0.814
0.804
0.549
1lama-2-7b
0.821
0.823
0.753
0.756
0.177
0.186
TL (-20)
Ilama 3.1-8B
0.831
0.831"
0.833
0.835
0.559
0.567
Ilama 3.2-3B
0,801
0.802
0.741
0.744
0.178
0.166
aya-expanse-Sb
0.784
0.788
0.757
0.758
0.480
0.491
Aug
0.809
0.771
0.348
Ilama-2-7b
0.745
0.742
0.381
0.375
0.281
0.288
TL (-24)
Ilama 3.1-8B
0.806
0.808
0.787
0.790
0.708
0.717
Ilama 3.2-3B
0.635
0.632
0.311
0.307
0.420
0.420
aya-expanse-8b
0.733
0.731
0.619
0.615
0.576
0.589
Aug
0.730
0.525
0.496
SemEval
Task
0.799
0.738
0.685
Best
Table 6: Spearman's 's (p) and Pearson's 's (r) correlation scores obtained with ALOPE framework for the monolingual regression task of emotion intensity prediction across multiple language pairs. The last row in the table shows the best Pearson correlation score obtained in SemEval- 2018 for the emotion-intensiiyy score prediction task (Mohammad et al., 2018). The bolded values show the highest Spearman score obtained and asterisks (*) indicate the highest Pearson score obtained.
19


Reviews:
[{"review_id": "GsTbMsvKPG", "reviewer": "Reviewer_m5bM", "summary": "This paper presents a technique for performing LLM-based QE as a regression task, whereas the “path of least resistance” with an LLM would be to just format it as another sequence generation task. Their technical contribution is a handful of techniques for adding regression heads at intermediate layers. They present results with several LLMs with parameter counts up to 8B and evaluate on several low-mid resource language pairs.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The work presents an elegantly simple technique that seems to work.\n- The authors promise to release their models and code.", "reasons_to_reject": "- There are few ablations. It would be useful to see more comparisons with different configurations on a single model (more choices of hidden layer, different pooling strategies for the hidden states, etc.), rather than comparisons using many models whose results are usually poor or unremarkable (such as the Aya model, for example).", "questions_to_authors": "- It seems to me that the adaptation of the models for regression is completely orthogonal to the use of LoRA. Is there some connection between these two factors that I’m missing, or would it be capable to do the regression without LoRA layers as well?\n- It seems like the most obvious place to attach the regression head would be after the final layer, rather than after some intermediate layer. Was this tried and did it work?\n- A lot of space in Section 3 could be saved by omitting equations that are well-known, or putting them inline. For example, including the softmax equation (L174) is unnecessary.\n- Some of the comparisons are presented in an unclear way. For example, Table 1 does not include SIFT even though it’s the most important baseline and should not have been demoted to an appendix. On the other hand, the plots in Figure 3 compare dynamic-weighting to multi-head regression to SIFT but not to vanilla ALOPE, which is the most natural thing to compare to (unless I’m misunderstanding something).\n- The formulation of dynamic weighting allocates one parameter per model layer, and this weight does not depend on the model input. Therefore, after training the weighting of layers is the same for all inputs (perhaps static weighting would have been a better name). Did you also try techniques for learning the weights that do depend on the input in some way? This could be done, for example, by computing the weight for each layer as a linear project of h_k.", "extra_scores": {}}, {"review_id": "fNnUYKDboP", "reviewer": "Reviewer_6xjS", "summary": "This paper has proposed ALOPE, an adaptive layer-optimization framework that is tailored to improve LLM-based quality estimation for machine translation tasks by facilitating optimal cross-lingual representations. More concretely, the authors proposed the layer-specific adaption in Transformer models, and two strategies of dynamic weighting and multi-head regression to improve the quality estimation prediction. Experimental results shows that the proposed framework shows improvement over LLM-based QE approaches.\n\nExperimental setup is a little unclear, and the manuscript could be restructured. The paper lacks of model comparison, just being assessed with various LLMs, which is okay but not sufficient enough to claim its effectiveness.\nIn Table 1, Aya model tends to show smaller correlation score. Do you have any comment on this? Instead of instruction-based finetuning, have you ever apply reinforcement learning from human feedbacks, which could be another baseline for comparison.\nThe paper could be updated for better readability:\n- The motivation behind the layer optimization is unclear in the first section. Could you explain more carefully why you decided to focus on layer optimization and what to be expected with the ALOPE framework?\n- Figure 2 looks a bit confusing. Can you try to simplify blocks as much as possible?\n- Section 3.1 could move to Section 3.3. It is hard to follow the experimental settings in the current order.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper proposed a novel approach of adaptive layer optimization for translation quality estimation in the LLMs. Applicable to any LLMs.", "reasons_to_reject": "- The paper could be revised for better readability.\n- Lack of baselines or other models for system comparison, despite some being mentioned in Section 2.", "questions_to_authors": "See the summary.", "extra_scores": {}}, {"review_id": "sJ7iA8Dlb8", "reviewer": "Reviewer_ffUc", "summary": "ALOPE introduces an adaptive layer-optimization framework that enhances Large Language Models' capabilities for Machine Translation Quality Estimation through layer-wise adaptation with regression task headers. It does it in a parameter-efficient way, leveraging Quantized Aware Training plus LORA adapters (QLORA).\n\nThe work demonstrates that intermediate Transformer layers provide superior cross-lingual representations for QE tasks compared to final layers, and introduces dynamic weighting and multi-head regression strategies that further improve performance across eight low-resource language pairs.\n\nALOPE represents a novel contribution that bridges the gap between generative LLMs and regression-based QE tasks, showing clear improvements over existing LLM-based approaches. The paper is well-structured and comprehensive, with thorough empirical evaluation across multiple models and language pairs.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper systematically identifies which Transformer layers in LLMs are most effective for cross-lingual QE, finding that intermediate layers (particularly TL-7/TL-11) consistently outperform the final layer, a counter-intuitive but valuable insight.\n* ALOPE demonstrates practical improvements over standard instruction fine-tuned LLMs across all evaluated low-resource language pairs, achieving results comparable to (and in some cases exceeding) state-of-the-art encoder-based QE models.\n* The approach is model-agnostic and computationally efficient through LoRA adaptation, making it highly applicable to existing LLM deployments without extensive reconfiguration.\n* The proposed framework allows for scaling existing LLM-based MT systems with QE capabilities and will be released as open-source, enhancing reproducibility and future research.", "reasons_to_reject": "* While the paper demonstrates improvements over other LLM-based approaches, ALOPE still doesn't consistently outperform state-of-the-art encoder-based QE models like COMET across all language pairs.\n* The ablation study exploring generalizability to monolingual regression tasks is somewhat limited, making it difficult to fully assess whether the findings about layer-specific adaptation transfer to other regression scenarios.\n* The paper could more thoroughly explore why En-Ta language pair consistently shows different patterns compared to other language pairs, potentially missing deeper insights about linguistic factors affecting cross-lingual QE.\n* While the dynamic weighting and multi-head regression strategies show promise, the improvements over layer-specific adaptation are relatively modest, suggesting these approaches might need further refinement.", "questions_to_authors": "## Questions\n* Have you explored whether the optimal layer findings (TL-7) generalize to larger LLMs beyond the 8B parameter models tested? Would you expect similar patterns in models with significantly more layers? Do you anticipate these findings to generalize to other model families?\n\n* The paper demonstrates that ALOPE achieves comparable results to encoder-based QE models despite using causal LLMs. What are the computational and memory trade-offs compared to these encoder-based approaches?\n\n## Suggestions/Comments\n* Can you add a discussion on why predicting a scalar for MT is necessary with LLMs? Does it matter anymore? QE was framed as a way to determine which translations required human inspection. Doesn't it make sense to move towards an error extraction (a la MQM) which seem more useful to detect errors, and surface them to humans?\n* The comparison with encoder-based models in Table 2 could be expanded to include more details about model sizes and training data requirements.\n* The plots in Figure 3 would benefit from more distinct colors for better readability, especially when printed in grayscale. \n* Consider adding more discussion about the practical applications and deployments of ALOPE in real-world MT systems.\n\n## Missing Citations\n* Llama 3 family herd of models\n* Recent work on layer-wise knowledge probing in multilingual LMs such as Dufter and Schütze (2020) \"Identifying Elements Essential for BERT's Multilinguality\"\n* Fan et al. (2023) \"Layer-selective Rank Reduction for Parameter-efficient LLM Adaptation\"", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper looks at Quality Estimation (QE) using an LLM for Machine Translation. The method is described by one reviewer as “elegantly simple that seems to work”. The method is an adaptive layer-optimization framework which is interesting algorithmically and novel. Reviewers made positive comments such as “The paper systematically identifies which Transformer layers in LLMs are most effective for cross-lingual QE,” and “approach is model-agnostic and computationally efficient through LoRA adaptation, making it highly applicable to existing LLM deployments without extensive reconfiguration”.

Multiple reviewers noted that it was applicable to any LLM.

The authors addressed multiple reviewer concerns and scores and all reviewers updated (positively) their reviews afterwards. This appears to have the potential for a lot of interest in the multilingual and machine translation community at COLM. 

Overall, all of the reviewers had positive views of the paper and it appears to be technically sound and interesting to the community.
Decision: Accept