RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
B.2 Details on Finetuning
When finetuning NLLB-200, we use a batch size of 16k target tokens.
For bilingual systems, we train for 30 updates.
When training multilingually, we train for 60 updates.
We use a learning rate of 0.0001 with an inverse squared root schedule and 20 warmup steps.
Dropout is set to 0.1 .
Attribute Classifier Training
The classifier operates on meanpooled decoder hidden states and consists of two feedforward layers with ReLU activation in between.
The first layer projects from the 1024 Transformer hidden dimension to 256 ,
the second layer from 256 to C , the number of attribute classes.
In our experiments, C is 2 for formality control (formal, informal)
and 3 for gender control (feminine, masculine, neutral) 12 12 12 As our test set only covers two genders, we only report scores on two genders. .

We train the classifier on a frozen NLLB-200 600M model with an effective batch size of 32 k target tokens.
The learning rate is 0.002 with an inverse square root schedule and 20 warm-up steps.
We use the Adam Kingma and Ba ( 2015 ) optimizer with betas of (0.9,0.98) .
Dropout and label smoothing are set at 0.1 .
For formality control, we train the monolingual classifiers for 100 updates and multilingual for 250 updates.
For the gender control, we train for 25 updates due to the small dataset and simplicity of the training data.
Hyperparameters
For the classifier guidance hyperparameters,
on the en \rightarrow de training data of CoCoA-MT,
we searched among step size [0.05,0.1,0.5] ,
and number of iterations [3,5] .
We used 5 iterations and 0.1 step size for formality control,
and 5 iterations and 0.05 step size for grammatical gender control.
We do not use KL regularization and postnorm fusion as in Dathathri et al. ( 2020 ) ,
since they degraded performance in initial experiments.
Decoding Speed
Decoding with our approach is slow due to the repeated gradient updates.
For instance on formality control,
decoding on the test sets of 600 sentences takes around 30 minutes.
Figure 1: The number of translation directions with attribute-annotated data ( right ) is far less than that of what massively pretrained models serve ( left ).; The number of translation directions with attribute-annotated data ( right ) is far less than that of what massively pretrained models serve ( left ).
Figure 2: Left: Inference -time control by gradient-based classifier guidance: training classifiers for attributes on decoder activations,
and using its predictions to edit inference-time model activations towards desired attributes.
Right: Standard training -time control by finetuning on attribute-specific data.; Left: Inference -time control by gradient-based classifier guidance: training classifiers for attributes on decoder activations,
and using its predictions to edit inference-time model activations towards desired attributes.
Right: Standard training -time control by finetuning on attribute-specific data.
Data overview. Codes: German (de), Spanish (es), French (fr), Hindi (hi), Italian (it), Korean (ko), Portuguese (pt), Russian (ru), source (src), target (tgt).
Formality control results in supervised condition (controllers trained on formality-annotated data).
Grammatical gender control results in supervised condition (cross-domain: controller trained on gender-annotated data from a different domain).
Zero-shot formality control results. Best and second best results under the same data condition are marked.
Zero-shot grammatical gender control results on new target languages with domain mismatch .
Human evaluation on Bengali,
with quality on a 5-point scale \uparrow and formality on a 3-point scale ( \uparrow : formal) with standard deviations .
Last two columns show pairwise comparison of formality scores to baseline NLLB-200 given the same source sentences (winning: scoring more in the direction of the desired formality).
Zero-shot formality control results on new source languages, using controllers trained on English as source.
Sources are {de, es, fr, hi, it}.
Colored columns indicate source formality agreeing with desired target formality.
Zero-shot grammatical gender control results on new source and target languages.
Overview of OPUS-100 data we used to train the Transformer-base.
Details on domain mismatch training setup.
Comparison to prior works with different data conditions.