Introduction
Pretrained multilingual translation models with massive coverage Zhang et al. ( 2020 ); Liu et al. ( 2020 ); Fan et al. ( 2021 ); Xue et al. ( 2021 ); NLLB Team et al. ( 2022 ) have become of the backbone of many translation systems.
While their off-the-shelf translation quality has been constantly improving Fan et al. ( 2021 ); Ma et al. ( 2021 ); NLLB Team et al. ( 2022 ) ,
the flexibility of customization towards desired attributes, such as formality or grammatical gender, is another important metric.
Adapting generic systems for attribute-controlled translation relies on training data with attribute information.
Creating such annotated data often requires language-specific knowledge and manual curation.
This makes data acquisition challenging even for single languages.
When scaling to the numerous directions served by massively multilingual models, it quickly becomes impractical, as shown in Figure 1 .
While prior works Michel and Neubig ( 2018 ); Saunders et al. ( 2020 ); Nadejde et al. ( 2022 ) showed promising results of finetuning on limited attribute-annotated data,
to allow other languages without supervised data to similarly benefit from the customization possibilities,
the transferability of the attribute controllers remains to be studied.

A straightforward way to achieve attribute control is finetuning on attribute-specific data.
Recent works Rippeth et al. ( 2022 ); Wu et al. ( 2023 ) have shown that finetuning with just hundreds of attribute-specific sentences is sufficient.
However, small finetuning data also brings the risk of overfitting and catastrophic forgetting Freitag and Al-Onaizan ( 2016 ); Thompson et al. ( 2019 ) .
It is especially relevant when generalizing to new languages,
where finetuning on some languages may erase the knowledge of others from pretraining Garcia et al. ( 2021 ); Cooper Stickland et al. ( 2021 ); Liu and Niehues ( 2022 ) .
While these issues may be mitigated by partial finetuning Houlsby et al. ( 2019 ); Bapna and Firat ( 2019 ) ,
domain mismatch between the finetuning data and the test domain can still degrade translation quality.
We will validate these concerns in various zero-shot conditions with different language relatedness and domains.

On the other end of the spectrum,
inference-time customization is another paradigm of attribute control.
In this case, the pretrained model is fully unchanged in the training stage.
At inference time, the generation process is steered towards desired attributes by
e.g. re-weighting entries in the output distribution Saboo and Baumann ( 2019 ); Yang and Klein ( 2021 ); Landsman et al. ( 2022 ) or editing model activations Dathathri et al. ( 2020 ) .
To enable cross-lingual transfer, the controller must be trained on features that are shared across languages.
This precludes methods that operate on the surface vocabulary level. In this work,
we will extend an activation-based approach Dathathri et al. ( 2020 ) originally for decoder-only models to cross-lingual transfer on pretrained translation models.

Task Formalization We focus on the following task:
Given a pretrained many-to-many multilingual translation model covering N languages and N(N-1) translation directions, along with parallel data on k ( k\ll N(N-1) ) translation directions where the target translation corresponds to specific attributes (e.g., formality level),
we aim to customize the pretrained model to translate with desired attributes for as many directions as possible.
We refer to the subsequent model as an attribute controller .
Specifically,
after learning on the k sets of parallel data with attribute annotation,
to what extent can we transfer the attribute controller to the remaining N(N-1)-k translation directions?