RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Background and Related Work
Attribute-Controlled Translation Previous works investigated controlling various attributes of machine translation outputs, for instance politeness Sennrich et al. ( 2016 ); Niu et al. ( 2018 ); Feely et al. ( 2019 ) , gender Vanmassenhove et al. ( 2018 ); Saunders et al. ( 2020 ) , length Takase and Okazaki ( 2019 ); Lakew et al. ( 2019 ); Marchisio et al. ( 2019 ); Niehues ( 2020 ) , or style in general Michel and Neubig ( 2018 ); Schioppa et al. ( 2021 ); Vincent et al. ( 2023 ); Wang et al. ( 2023 ) .
As existing works mainly focus on supervised conditions with at least some supervised data, how these approaches generalize to new languages remains unclear.
In face of data scarcity, one approach is to use synthetic data by pseudo-labeling Rippeth et al. ( 2022 ); Lee et al. ( 2023 ) .
In our work,
by building upon massively multilingual translation models,
we do not assume the scalability of creating synthetic data for all languages served by the backend model, nor do we assume a classifier that can a priori distinguish attribute classes for zero-shot languages.

Multilinguality for Controllable Generation Our work is also related to controllable text generation in general.
Despite steady progress in this field Keskar et al. ( 2019 ); Krause et al. ( 2021 ); Yang and Klein ( 2021 ); Liu et al. ( 2021 ) ,
how the controller generalizes across languages is likewise less explored.
With the recent surge of large language models (LLMs), attribute-controlled translation has also been addressed by prompting multilingual language models in a few-shot manner Sarti et al. ( 2023 ); Garcia et al. ( 2023 ) .
Notably, Sarti et al. ( 2023 ) reported promising few- and zero-shot attribute control results using multilingual LLMs.
In this work,
we take a different perspective by using a pretrained dedicated encoder-decoder translation model as backend,
and transferring the attribute control capabilities with lightweight add-ons.
As currently open LLMs still lag behind dedicated translation models Zhu et al. ( 2023 ); Sarti et al. ( 2023 ) especially on low-resource languages Robinson et al. ( 2023 ) ,
we believe improving the attribute control capabilities of massively multilingual conventional models is still highly relevant.

Multilingual Domain Adaptation Attribute control can be viewed as a light domain adaptation task.
Prior works Cooper Stickland et al. ( 2021 ); Vu et al. ( 2022 ) adapting pretrained multilingual models have reported catastrophic forgetting of languages absent from the finetuning stage.
Our results on finetuning for zero-shot attribute control (§ 6.1 ) shows a different picture.
One potential reason is that, compared to adapting to fully new domains such as medical or law texts,
the attribute control task can be learned with less data.
This in turn requires less intense finetuning and is therefore less vulnerable to forgetting.
Transferring Attribute Controllers for Multilingual Translation
To generalize to new translation directions,
an ideal controller should be attribute - rather than language -specific.
That is,
its representation for different attribute labels varies little with specific languages.
Finetuning-Based Control:
A more common way to realize control is finetuning the pretrained model on attribute-specific parallel data, as done in domain adaptation Freitag and Al-Onaizan ( 2016 ) .
To transfer to directions without annotated data,
the adaptation step must mostly learn the desired attributes rather than the specific languages in finetuning,
so as not to forget the languages without annotated data.
On our tasks, naive finetuning already works effectively:
We finetune the full model on each attribute,
resulting in one specialized model per attribute as shown in Figure 2 . 4 4 4 We tried prepending attribute tags to the source sentences Chu et al. ( 2017 ); Kobus et al. ( 2017 ) ,
but this was not enough to make the pretrained model to be attribute-aware.
A potential reason is that the pretrained model tends ignore the source tags as noise, and that the low amount of finetuning data cannot re-establish the importance of the tags. Partial finetuning e.g. with adapters Bapna and Firat ( 2019 ); Philip et al. ( 2020 ) is a more parameter-efficient approach.
We do not explore partial finetuning in this work,
as it does not fully align with our focus on the transferability of attribute controllers.
Experimental Setup
We experiment on two attribute control tasks: formality and grammatical gender control.
As outlined in Table 1 , the training data has English on the source side.
For the target languages, there is one set of translations per attribute.
The low data volume not only reflects the practical challenge of data acquisition, but is also an established condition in existing benchmarks Nadejde et al. ( 2022 ) .
Formality Control (In-Domain)
The training data come from CoCoA-MT Nadejde et al. ( 2022 ) 5 5 5 We excluded Japanese,
where our pretrained model has very low translation accuracy on formality-annotated words ( < 40%, whereas all 5 other languages score > 60%). ,
where the test domain overlaps with training.
For zero-shot conditions, we transfer controllers trained on different language pairs to new translation directions.
Specifically, we investigate the following two cases:
Transfer to New Target Languages
We train the attribute controllers on one or multiple target languages to assess the impact of multilinguality on transfer.
We compare the following settings:

Single-direction : We use en \rightarrow es and de as representative Romance and Germanic languages;

Multilingual : We train on all languages in the training data: en \rightarrow { de , es , fr , hi , it }.

For the new target languages, we choose three directions from the IWSLT 2023 formality control shared task 6 6 6 https://github.com/amazon-science/contrastive-controlled-mt/tree/main/IWSLT2023 Agarwal et al. ( 2023 ) : en \rightarrow pt ( close ), en \rightarrow ru ( related ),
and en \rightarrow ko ( distant )
for their different degrees of relatedness to the languages in training.
Among them, en \rightarrow ko has 400 sentences of supervised data.
We use it to establish the oracle performance in the presence of supervised data.
Transfer to New Source Languages
We re-align the CoCoA-MT test set using English as pivot,
creating a new test set with non-English source and target sentences. 7 7 7 The original test sets only have English input. As the English sentences mostly overlap,
we create new pairs of two non-English languages by matching their English translations. Unlike translating from English,
here the source sentences also contain formality information.
This allows testing if the model can: 1) preserve the source formality level; 2) change the source formality level when steered so.
Gender Control (Out-of-Domain)
For the formality control setup above,
the data for training the attribute controller come from the same domain as the test set.
To evaluate domain generalization,
for grammatical gender control,
we train the controller on texts with very different styles from the test data.
For training the attribute controller,
we use the en - es set from Saunders et al. ( 2020 ) 8 8 8 https://github.com/DCSaunders/tagged-gender-coref#adaptation-sets with artificial sentences of very simple grammatical structure up to 7 words.
In contrast,
for the test set
we use MuST-SHE Bentivogli et al. ( 2020 ) ,
which consists of TED talks with much longer sentences and more versatile styles.
More dataset details are Appendix A.2 .
Besides transfer to new target languages like previously (§ 4.1 ), we also explore the following setting:
Transfer to New Source & Target Languages
The MuST-SHE test set comes in en -{ es , fr , it }.
Like previously, we re-align them using English as pivot, creating non-English source and target sentences.
In this case, both the source and target sentences have the same gender.
As the attribute training data is in en \rightarrow es ,
we evaluate { es , fr } \rightarrow it and { es , it } \rightarrow fr for the transfer to new translation directions where both the source and target languages differ from training.
Models and Evaluation
Models We use two types of backend models.
For the main experiments,
we use the pretrained NLLB-200 distilled 600M model NLLB Team et al. ( 2022 ) , which covers 200 languages for many-to-many translation.
We also train a Transformer-base Vaswani et al. ( 2017 ) from scratch to verify if observed phenomena are specific to models with massive multilingual pretraining.
The Transformer-base model covers all languages in our experiments and is trained on OPUS-100 Zhang et al. ( 2020 ) .
Details of these data are in Appendix A.1 .
Training and inference details are in Appendix B .

Control Evaluation For formality control, we report matched accuracy (M-Acc; %) following Nadejde et al. ( 2022 ) .
For gender control, we use the official evaluation script Bentivogli et al. ( 2020 ) for accuracy (%).
For formality, as the test set is the same for both formalities, the baseline M-Acc for the two formality labels add up to 1.0.
This is not the case for gender control.

Quality Evaluation We use COMET \uparrow Rei et al. ( 2020 ) 9 9 9 with Unbabel/wmt22-comet-da ( \times 100 for readability) as the main translation quality metric,
and additionally report BLEU \uparrow 10 10 10 using sacreBLEU Post ( 2018 ) with confidence intervals: bs:1000—rs:12345—c:mixed—e:no—tok:13a—s:exp—v:2.3.1 to compare to prior works.
Note that BLEU is impacted by n -gram matches on the correct formality or gendered words,
while COMET is less susceptible to the artifact.
For COMET score comparisons, we run paired T-tests and bootstrap resampling using comet-compare .
We use "*" or "†" to mark systems better or worse than the base pretrained model at p=0.05 .

Human Evaluation To test the transfer to real low-resource languages,
we conduct a human evaluation on Bengali, which was marked as low-resource in the NLLB-200 training data NLLB Team et al. ( 2022 ) .
Details on the evaluation are in Appendix C .

Baselines Few existing works experimented on the same data conditions as ours.
An exception is the “ mBART-large Gold Finetuned ” model by Rippeth et al. ( 2022 ) ,
who finetuned mBART Liu et al. ( 2020 ) on parts of CoCoA-MT Nadejde et al. ( 2022 ) for formality control.
Their results overlap with our
supervised results on en \rightarrow {de, es, hi} and
zero-shot results on en \rightarrow ru .
Other than this,
the majority of prior works used more relaxed data conditions than ours,
e.g.,
using an existing attribute classifier that covers zero-shot languages for pseudo-labeling Lee et al. ( 2023 ) or hypothesis reranking Wu et al. ( 2023 ) .
We report these results in Appendix D .
Overall, our model’s performance is comparable to the leading systems.
Supervised Conditions
Table 2 and Table 3 show formality and gender control results respectively with supervised controllers on NLLB-200.
Overall, both finetuning and CG are able to steer the output towards given attributes, while maintaining the original translation quality or at the cost of a slight degradation.
Finetuning more effective than classifier guidance in supervised conditions:
A comparison of scores in Table 2 and Table 3 clearly shows FT is more effective than CG.
For formality control, FT consistently scores nearly 100% M-Acc.
It also substantially improves the quality scores due to adapting towards the specific domain of the attribute-annotated data,
which is the same as the test domain in this case.
On the other hand for CG,
while it also improves the formality accuracy,
the scores lag behind finetuning in both accuracy and quality.
The gap is especially prominent on hi and it , where the underlying NLLB model has a strong bias towards a single formality:
the accuracy for the rare formality is nearly zero (3.3% and 3.2% respectively).
This is likely to do with NLLB’s training data, which might be skewed towards one single formality for some languages.
In this case, CG can only partly recover the ability to generate translation in the formality NLLB is unfamiliar with.
These results indicate that CG is only effective when the underlying model does not suffer from an absolute bias towards one attribute.
Classifier guidance more robust to domain mismatch:
As motivated in § 4.2 ,
the gender control results in Table 3 allow us to assess the impact of domain mismatch between the controller training data and the test data, a very realistic scenario in practice.
Here, while finetuning achieves higher accuracy for gendered words, it also degrades translation quality by 0.9 COMET.
This provides further evidence that the previously improved COMET scores ( Table 2 ) are results of finetuning on in-domain data.
In contrast, the translation quality with CG does not significantly differ from NLLB by the T-tests, suggesting its stronger domain robustness.
We hypothesize it is because CG operates on the last decoder layer’s hidden states, which are just one projection away from the output vocabulary.
These representations likely contain more word-level than domain information,
which is precisely needed in the task of attribute control.
New Target Languages
Now we transfer the trained controllers to target languages unseen when training the controllers,
i.e., those without attribute annotation.
In Table 4 and Table 5 , we report the results on formality and gender control respectively.
In Table 4 , we also compare the single-direction and multilingual controllers as motivated in § 4.1 .
Gap between finetuning and classifier guidance shrinks in zero-shot conditions:
While finetuning was consistently leading in supervised conditions (§ 5 ),
now under zero-shot conditions with unseen target languages, the gap shrinks.
For formality control, on Korean, the most distant language,
CG consistently achieves stronger control results than finetuning,
indicating more robustness when transferring to unfamiliar settings.
Overall in Table 4 ,
for the main experiments on NLLB-200,
CG outperforms FT in 7 of the 9 pairwise comparisons ({ de , es , multi } \times 3 target languages).
With gender control results in Table 5 ,
finetuning achieves stronger control accuracy (avg. + 4.9% abs.)
but degrades translation quality ( - 0.6 COMET) due to domain mismatch.
On the other hand, CG retains the translation quality.
This confirms the previous finding (§ 5 ) on its stronger domain robustness.
Multilingual controllers help when the base model is not massively multilingual:
In Table 4 , controllers trained on multiple translation directions ( multi ) are compared to those trained on single directions ( en \rightarrow es or de ).
On Transformer-base, multi consistently outperforms its single-direction counterparts, regardless whether the controller is finetuning- or CG-based.
In contrast,
for the pretrained NLLB,
there is no clear distinction between the multilingual systems and rest.
This indicates that NLLB does not further benefit from multilinguality in the controller training stage,
likely because it already underwent a massively multilingual pretraining stage.
This shows that massively multilingual models are a useful basis for attribute control especially when annotated resources are limited to single languages.
Classifier guidance is complementary with finetuning:
When applying CG on top of the finetuned models, we see the strongest control accuracy for both formality and gender control.
This observation is consistent whether the base model is the pretrained NLLB or the normal Transformer-base.
Compared to finetuning alone, the addition of CG also does not degrade translation quality on NLLB.
On the more challenging case of gender control which involves domain mismatch,
adding CG to finetuning does not impact translation quality on fr and causes a slight degradation on it .
This is likely linked to poor hyperparameter choices in CG:
due to time constraints we directly used the hyperparameters when applying CG alone,
which are too strong for models already finetuned for attribute control.
We are optimistic for improved scores under more fitting hyperparameters.
Finetuning did not erase knowledge on other languages:
To our surprise and different from results in domain adaptation Cooper Stickland et al. ( 2021 ); Vu et al. ( 2022 ) ,
finetuning did not erase the pretrained model’s knowledge on the target languages absent in supervised finetuning,
as reflected by the translation quality scores ( Table 4 , 5 ).
This is not specific to NLLB,
but also observed on the Transformer-base trained with random initialization on a few translation directions.
Therefore,
this phenomenon is not a result of massively multilingual pretraining,
but more likely linked to the light finetuning strength with limited number of updates and small learning rates.
Comparison to oracle data condition:
In the bottom rows of Table 4 , we report the oracle performance of using 400 sentences as supervised data for training the controllers.
Our strongest zero-shot results match the performance of oracle CG, but still lag far behind the upper-bound of finetuning on in-domain data with attribute annotation (oracle FT).
We believe this gap is magnified as Korean is not only linguistically distant from the languages used in training, it also differs in the notion of formality:
Korean involves multiple levels of formality instead of a binary informal-formal distinction.
For the zero-shot transfer, this means transferring a controller trained for binary control to a multi-class problem with an unknown class mapping, which is naturally more challenging.
Human Evaluation on Bengali:
The results are in Table 6 .
First, adding attribute control does not appear to impact translation quality.
Second, pairwise comparisons with the baseline show both CG and finetuning are effective in formality control, where CG has slightly higher win ratio than FT against the baseline.
Third, the impact on formality scores is more prominent when steering towards informal translation.
This likely because the baseline translations already have a high level of formality.
Moreover,
the rare usage of the lowest formality level in Bengali (Appendix C ) could explain the relatively high formality scores for the systems steered towards “informal” (rows (3) and (5) ).
New source languages easier than new target languages:
In Table 7 , we report the results of transferring controllers trained with English source to new source languages.
Contrasting these scores with the target-side zero-shot results in Table 4 ,
it is clear that transferring to new source languages is a much easier task.
This is expected, as attribute-controlled translation primarily places lexical constraints on the target side.
Once the controller can generate translations with the correct attribute, swapping the source language does not pose a large challenge.
Even when the source formality disagrees with the desired output formality (uncolored columns in Table 7 ),
the controllers are able to steer the translations toward the required attributes.
NLLB struggles to preserve source attributes:
Contrasting the colored “base” cell in Table 7 with its uncolored counterpart,
we see that NLLB does have some notion of formality in the source sentences,
as source sentences with the correct formality improves accuracy on the desired formality (57.8 vs. 42.2% and 79.7 vs. 20.3%).
However,
the signals in the input alone are insufficient for generating the correct formality.
This is confirmed by another zero-shot experiment when both the source and target languages are new ( Table 8 ).
Here the sources already contain the correct grammatical genders.
Despite this, NLLB cannot fully utilize the signals in the source, especially on the feminine gender.
Its accuracy (76.1-83.2%) still lags behind the masculine class (87.0-90.4%).
Both CG and finetuning substantially improve the accuracy and mostly close the gap between the two grammatical classes.
This shows both approaches strengthen the source signals that are otherwise neglected.
Conclusion
To generalize attribute-controlled translation to data-scarce conditions,
we asked the question “how transferable are attribute controllers on pretrained multilingual translation model?”.
We use a novel classifier guidance method to extend a pretrained NLLB-200 model for attribute control and contrast its performance to finetuning-based control.

Our results led to the following recommendations for upgrading existing multilingual translation systems with attribute control capabilities: 1) Given in-domain target sentences annotated with attributes, even as few as the lower hundreds, finetuning is the primary choice. 2) In case of distant new target languages or strong domain mismatches between the attribute-annotated data and test data, decoding with classifier guidance is more promising. Otherwise finetuning is recommended. 3) In case specific resource constraints preclude finetuning or hosting multiple specialized variants of the underlying model, we then recommend inference-time control by classifier guidance. 4) In case the underlying translation model is not massively multilingual, finetuning the model or training the controller on multiple target languages is beneficial.
More Fine-Grained Attributes
Our classifier guidance approach works with discrete labels,
making it not directly applicable to use-cases with more fine-grained or continuous attributes.
In particular, although the gender classifier training incdlues a gender-neutral class,
in evaluation we were only able to test two genders,
limited by the availability of test data.
As more test datasets with fine-grained attributes become available,
our approach can be further improved and validated for these use-cases.
Inference Speed
Decoding speed is a main downside of our classifier guidance approach.
This is a result of multiple gradient-based updates of model activations at each decoding time step.
Despite the promising zero-shot results, further speed-up is necessary is make it realistic for deployed systems.
Acknowledgement
We thank Supriti Sinhamahapatra for help with the human evaluation, and the anonymous reviewers for helpful feedback.
This work was performed on the HoreKa supercomputer funded by the
Ministry of Science, Research and the Arts Baden-Württemberg and by
the Federal Ministry of Education and Research.
Part of this work was supported by funding from the pilot program Core-Informatics of the Helmholtz Association (HGF).
A.1 OPUS-100 Data for Transformer-Base
The data overview is in Table 9 .
For tokenization, we use the SentencePiece Kudo and Richardson ( 2018 ) model from NLLB-200 11 11 11 https://github.com/facebookresearch/fairseq/tree/nllb/#preparing-datasets-for-training NLLB Team et al. ( 2022 ) .
The model is trained to translate from and into English.
A.2 Details on Domain Mismatch Data
For the grammatical gender control experiments with domain mismatch (§ 4.2 ),
the training domain differs from the test sets in both style and length.
An overview is shown in Table 10 .

During training,
an example tuple of (input, output, attribute label) is:
("the actor finished her work.", "La actriz terminó su trabajo.", 0: feminine)
("the actor finished his work.", "El actor terminó su trabajo.", 1: masculine).
The training sentences are all artificial sentences following this simple subject-verb-objective structure.
This differs significantly from the test sets with public speaking texts.
Preprocessing
For CoCoA-MT Nadejde et al. ( 2022 ) ,
many test inputs contain multiple sentences.
When directly decoding, NLLB-200 NLLB Team et al. ( 2022 ) suffered from severe under-translation,
where the output translation only contains one sentence.
We therefore split the input by sentence boundaries and decode sentence by sentence.
Hyperparameters
When decoding, we use a beam size of 4 and length penalty of 1.0 .
Evaluation
To evaluate BLEU and COMET scores, we concatenate the hypotheses and references from different attributes.
It is also the case when reporting the multi-source results in Table 7 .