Enhancing Cross-lingual Prompting with Dual Prompt Augmentation∗
Year: 2023

Abstract:
Prompting shows promising results in fewshot scenarios. However, its strength for multilingual/cross-lingual problems has not been fully exploited. Zhao and Schütze (2021) made initial explorations in this direction by presenting that cross-lingual prompting outperforms cross-lingual finetuning. In this paper, we conduct an empirical exploration on the effect of each component in cross-lingual prompting and derive language-agnostic Universal Prompting, which helps alleviate the discrepancies between source-language training and target-language inference. Based on this, we propose DPA, a dual prompt augmentation framework, aiming at relieving the data scarcity issue in few-shot cross-lingual prompting. Notably, for XNLI, our method achieves 46.54% with only 16 English training examples per class, significantly better than 34.99% of finetuning. Our code is available at https: //github.com/DAMO-NLP-SG/DPA.

text: Prompting shows promising results in fewshot scenarios. However, its strength for multilingual/cross-lingual problems has not been fully exploited. Zhao and Schütze (2021) made initial explorations in this direction by presenting that cross-lingual prompting outperforms cross-lingual finetuning. In this paper, we conduct an empirical exploration on the effect of each component in cross-lingual prompting and derive language-agnostic Universal Prompting, which helps alleviate the discrepancies between source-language training and target-language inference. Based on this, we propose DPA, a dual prompt augmentation framework, aiming at relieving the data scarcity issue in few-shot cross-lingual prompting. Notably, for XNLI, our method achieves 46.54% with only 16 English training examples per class, significantly better than 34.99% of finetuning. Our code is available at https: //github.com/DAMO-NLP-SG/DPA.

heading: 1 Introduction
text: Although adapting Pre-trained Language Models (PLMs) (Devlin et al., 2019) to downstream NLP tasks via finetuning is the de facto mainstream paradigm under fully supervised settings (Wang et al., 2018), prompting1 (Gao et al., 2021; Radford et al., 2019; Brown et al., 2020; Schick and Schütze, 2021a,b) has demonstrated its superiority over finetuning in low-resource scenarios. Typically, prompting reformulates the classification task as a language modeling problem over manuallydesigned natural language prompts.
Despite the effectiveness of prompting on English tasks, its potential for cross-lingual problems, which assume the availability of the training data
∗This work was supported by Alibaba Research Intern Program. It was done when Meng Zhou was an intern at Alibaba. Xin Li is the corresponding author.
1In this work, the term “prompting” refers to prompt-based finetuning, where the parameters of PLMs are finetuned.
in high-resource languages (e.g., English) only, is still under-explored. Zhao and Schütze (2021) is the pioneering work to apply prompting to crosslingual NLP. However, their major efforts are spent on comparing different training strategies for crosslingual prompting such as discrete prompting and soft prompting. They do not fully investigate the design choice of key components in prompting, i.e., prompt template and verbalizer.
To provide a practical guide for designing cross-lingual prompting, we first conduct an empirical analysis to explore the effects of each prompting component on the performance of crosslingual transfer. Our preliminary study shows that template-free prompting combined with Englishonly inference, dubbed as language-agnostic “Universal Prompting” (UP) in this paper, generally performs well across different few-shot settings. Intuitively, UP avoids the discrepancies between the source-language training and the target-language inference, which intrinsically better fits crosslingual tasks.
The derived UP is a concise solution with reasonable performance but does not take advantage of other available resources in the context of multilingual problems, e.g., the translation of verbalizers in target languages. Motivated by this fact, we propose a Dual Prompt Augmentation (DPA) framework to alleviate the data scarcity issue in few-shot scenarios. Firstly, we introduce multilingual verbalizers as answer augmentation for prompting, where the translated label tokens are treated as additional target-language supervision. Secondly, we propose prompt mixup as prompt input augmentation, which mixes the prompt representations in each batch. Intuitively, given two prompt representations on real data, we can generate a virtual representation based on their interpolation, which encodes the semantics in between. Our DPA framework is not task-dependent and does not require either external unlabeled data (Xie et al., 2020) or
massive text manipulation efforts (Wei and Zou, 2019) compared with other data augmentation approaches.
In summary, our contributions are as follows:
• We develop language-agnostic Universal Prompting, a concise prompting baseline with competitive performance for cross-lingual transfer. • To overcome the data scarcity issue, we propose Dual Prompt Augmentation for cross-lingual prompting to perform data augmentation from the views of prompt answers and prompt inputs.

heading: 2 Language-Agnostic Universal Prompting
text: In this section, we first empirically investigate the importance of essential elements, i.e., template and verbalizer design, in cross-lingual prompting (Zhao and Schütze, 2021). Based on our investigation, we derive a more competitive baseline called Universal Prompting. It is language-agnostic because it does not make assumptions about the input language in template design, and the verbalizer during training is taken for all other languages. Note that, since soft prompting (SP) and mixed prompting (MP) rely on an external bidirectional LSTM (Hochreiter and Schmidhuber, 1997) to create soft prompts and do not outperform discrete prompting (DP) significantly, we mainly discuss DP in this work for a clear comparison.
As illustrated in Table 1, Zhao and Schütze (2021) directly utilize the translated templates and verbalizers for target-language inference, making templates and verbalizers language-dependent. However, the translated templates are not seen and the translated verbalizers are never modeled by the PLM during training. This leads to discrepancies between the source-language training and the target-language inference.
To alleviate such discrepancies, we consider three possible variants. Specifically, these three variants are derived by avoiding translation on the template and verbalizer tokens or removing the template words, see Table 1 for concrete examples.
We follow the experimental setup (refer to Section 4 for details) in Zhao and Schütze (2021) to evaluate the impact of the above designs 2. In Table 2, we observe that W/O TEMPLATE TRANS-
2As we employ a different evaluation method, the reproduced results of Zhao and Schütze (2021) are slightly different from the original ones. More details can be found in Section 4.
LATION achieves slight but stable improvements under different shots. W/O TEMPLATE WORDS simply removes the template words and achieves more obvious improvements. W/O VERBALIZER TRANSLATION 3 avoids using translation at the verbalizer end and brings in the most significant improvements. Therefore, by alleviating discrepancies either in the aspect of verbalizer or template, the performance of cross-lingual prompting can be further improved. By combining the advances of these variants, the Universal Prompting (UP) is derived to treat various languages in a unified fashion. Specifically, UP alleviates the discrepancy of prompt templates and verbalizers simultaneously, which is a much stronger baseline than Zhao and Schütze (2021) in multilingual tasks.
Note that the idea of removing template words in UP is distinct to “null prompt” (IV et al., 2021) from the perspective of motivation. “Null prompt” is proposed to simplify the manual prompt design on monolingual tasks. Compared with “null prompt”, the primary goal of UP is to alleviate the source-target discrepancies in cross-lingual transfer. Moreover, besides removing template words, our UP also involves the design choice for targetlanguage inference (W/O VERBALIZER TRANSLATION), which proves to be a larger contribution according to the empirical results shown in Table 2. The effectiveness of using the verbalizer in the source language is also found in (Lin et al., 2022).

heading: 3 Dual Prompt Augmentation
text: In prompting, the mask token is directly used for making predictions. In this section, we formalize a Dual Prompt Augmentation (DPA) framework based on this crucial element of prompting.

heading: 3.1 Prompt Answer Augmentation
text: In Section 2, we show that directly translating the verbalizers to the target language for inference is not helpful. In this subsection, we explore the usage of verbalizer translation at the training stage. Intuitively, their rich semantics could serve as highquality paraphrases (Jiang et al., 2021) of the English verbalizer and provide additional supervision to train multilingual models. Motivated by this, we define a multilingual verbalizer for the English
3Note that W/O VERBALIZER TRANSLATION refers to not applying translated verbalizers during inference. In Section 3 we will show how to exploit the translated verbalizers as answer augmentation during training.
training data, which can be regarded as answer augmentation for the mask token. Formally, given the pre-built prompt x filled with input sentences, the training objective is to maximize the likelihood of verbalized label tokens in multiple languages:
argmax θ ∑ x 1 |L| ∑ ℓ∈L logP ( ⟨mask⟩ = Vℓ(y)|x;θ ) (1)
where θ denotes the parameters of the PLM. Vℓ is the verbalizer in a certain language ℓ ∈ L, and it maps from the gold label to a specific word in language ℓ. 4 In comparison, UP only takes L = {EN}, which is a monolingual verbalizer.

heading: 3.2 Input Augmentation with Prompt Mixup
text: Previous mixup methods for NLP perform the whole-sequence interpolation at the input embedding level (Zhang and Vaidya, 2021; Guo et al., 2019) or hidden representation level (Jindal et al., 2020; Chen et al., 2020). However, directly applying previous methods to prompting has been shown
4Please refer to Appx. A for the language set we use
to even lead to a significant performance drop in Zhou et al. (2021). In prompting-based methods, the most important hidden space representation for classification is encoded at the position of mask tokens. Different training data may have different sequence lengths and their mask tokens are at different positions. The interpolation between the representation of a mask token and a normal verbal token would be meaningless in prompting. Therefore, we propose to interpolate between the top-most mask token representations to augment prompt inputs. Then the interpolated representation is fed into the masked language modeling head.
Formally, let mi = h(xi) and mj = h(xj) be the top-most hidden representations corresponding to the mask tokens of two prompts xi and xj , respectively. Then we perform linear interpolation to produce a virtual representation:
m̂ij = λh(xi) + (1− λ)h(xj) (2)
where λ follows a Beta distribution, i.e., λ ∼ β(α, α). The corresponding answer labels are linearly interpolated accordingly:
ŷij = λyi + (1− λ)yj (3)
Considering an augmented multilingual verbalizer as in Section 3.1, the training objective of this particular virtual example would be:
argmax θ
1 |L| ∑ ℓ∈L { λ logP ( ⟨mask⟩ = Vℓ(yi)|m̂ij ;θ ) +(1− λ) logP ( ⟨mask⟩ = Vℓ(yj)|m̂ij ;θ
)} (4) The interpolation is performed in a dynamic inbatch fashion. For a mini-batch drawn from the training set, we will split it into pairs and generate a virtual prompt representation based on each pair.

heading: 4 Experiments

heading: 4.1 Setup
text: Datasets We conduct experiments on two sentence-pair classification tasks: XNLI (Conneau
et al., 2018; Williams et al., 2018) for cross-lingual natural language inference and PAWS-X (Yang et al., 2019) for multilingual paraphrase identification. For these two datasets, while the evaluation data is human-translated, the golden training data is only available in English.
Evaluation We conduct our experiments by training the XLM-R base model (Conneau et al., 2020) on English. Then the model will be directly applied to other target languages, without using any training examples of the target language. To make a reasonable comparison between finetuning and prompting, we ensure finetuning to be better than a random guess on each language. Therefore, we randomly sample without replacement K ∈ {16, 32, 64, 128, 256} per class for XNLI and K ∈ {256, 512} per class for PAWS-X to construct the training set. Then we use the same number of shots from the validation split to select the best model (Perez et al., 2021).
The evaluation of few-shot cross-lingual transfer can be with large variance and depend on data selection (Zhang et al., 2021a; Zhao et al., 2021; Keung et al., 2020). In our work, to faithfully reflect the few-shot performance, separate training/validation sets are sampled for different runs.

heading: 4.2 Results
text: UP v.s. Finetuning/PCT On the XNLI dataset, even the simplest prompting method for crosslingual transfer, namely UP, consistently outperforms the finetuning (FT) method by a large margin. Besides, our language-agnostic UP also surpasses FT on the majority of languages on the more challenging PAWS-X. These observations suggest that prompting is indeed a better solution for few-shot learning in cross-language scenarios and our UP can serve as a strong baseline for crosslingual prompting. We also reproduce PCT (Qi et al., 2022), another recent cross-lingual prompting method based on data augmentation and consistency training, with our evaluation method. Table 3 shows that UP outperforms PCT consistently without any data augmentation approach or introducing additional loss terms.
Dual Prompt Augmentation With the proposed DPA framework, our prompting method achieves consistent improvement over UP, indicating that multilingual verbalizers from the answer view and prompt mixup from the input view are both effective ways to enhance cross-lingual prompting.
The comparison results in Table 3 and Table 4 also exhibit clear superiority of our method over crosslingual finetuning. Even in the most resource-rich settings, compared to FT, our method still obtains 7.1% (256 shots) and 4.9% (512 shots) absolute gains on XNLI and PAWS-X.
Ablation Study The performance of our prompting method will become worse when removing either prompt mixup or multilingual verbalizer, showing that both prompt input and prompt answer augmentation contribute positively to the improvement. We also notice that the negative effects brought by DPA W/O MV are generally larger, showing the necessity of target-language guidance for crosslingual prompting.

heading: 4.3 Inference Strategy
text: A natural extension for the DPA framework is to leverage the multilingual verbalizer in some way for target-language inference as well. For comparisons, we heuristically devise the following inference strategies :
(1) English Verbalizer The English verbalizer is still used when transferring to target languages. This strategy is used to produce results in Table 3 and 4. To formalize:
ŷ = argmax y
P ( ⟨mask⟩ = VEN (y)|x;θ ) (5)
(2) Target Language Verbalizer The verbalizer in the corresponding target language is used, which is the practice of Zhao and Schütze (2021) during inference time. However, in this case, our DPA framework has already modeled these words during the training time. To formalize:
ŷ = argmax y
P ( ⟨mask⟩ = Vtarget(y)|x;θ ) (6)
(3) Taking Maximum over the Multilingual Verbalizer In this strategy, we will take the maximum probability over the whole multilingual verbalizer. To formalize:
ŷ = argmax y,ℓ
P ( ⟨mask⟩ = Vℓ(y)|x;θ ) (7)
(4) Taking Sum over the Multilingual Verbalizer In this strategy, we will take the sum of probability over the whole multilingual verbalizer. To formalize:
ŷ = argmax y ∑ ℓ∈L P ( ⟨mask⟩ = Vℓ(y)|x;θ ) (8)
(5) Bilingual Verbalizer In this strategy, we will take the sum of probability over the target language verbalizer and the English verbalizer. To formalize, the predicted label ŷ is given by:
ŷ = argmax y
{P ( ⟨mask⟩ = VEN (y)|x;θ ) +P ( ⟨mask⟩ = Vtarget(y)|x;θ ) } (9)
We use the checkpoint of XLM-R trained by 128 shots on the XNLI dataset and make inference with different strategies. Table 5 shows the accuracy by employing different inference strategies. We show that with our DPA framework, the inference is quite robust to the utilization of the verbalizer. This can probably be attributed to answer augmentation via multilingual verbalizers, which help to model label
tokens in multiple languages. We choose to simply employ English-only inference due to its simplicity and slightly better performance to produce results in Tables 3 and 4.

heading: 5 Conclusion
text: In this paper, we first derive language-agnostic Universal Prompting, a concise but competitive baseline for cross-lingual prompting. The proposed DPA framework can further enhance cross-lingual prompting as shown on two sentence-pair classification tasks. In the future, we will consider verifying the effectiveness of prompting and the DPA framework in cross-lingual sequence tagging or question-answering tasks (Xu et al., 2023).

heading: 6 Limitations
text: Our work mainly focuses on cross-lingual sentencepair classification tasks. While it is directly applicable to single-sentence classification tasks (Li et al., 2020; Ye et al., 2020) but may require additional efforts to adapt our DPA framework to more complex cross-lingual tasks such as sequence tagging (Liu et al., 2021; Zhou et al., 2022, 2023; Zhang et al., 2021b) or question answering (Xu et al., 2022, 2023). Another limitation is that the proposed multilingual verbalizer in the DPA framework requires an external machine translator to produce the translated verbalizers. Finally, we limit the language set of the multilingual verbalizer to the set of target languages in a multilingual dataset. Extending this language set might give us greater improvement for cross-lingual tasks.

heading: A Additional Implementation Details
text: Implementation Package Our implementation is based on PyTorch (Paszke et al., 2019) and Huggingface Transformer (Wolf et al., 2019) framework.
Model Details XLM-R base model, containing 270M parameters, is pretrained on 2.5TB of filtered CommonCrawl on 100 languages. It contains 12 Transformer layers with hidden space dimensions of 768 and 12 attention heads in each layer.
Computing Infrastructure All of our experiments are conducted on a single Tesla V100-SXM2 32G. Gradient accumulation steps of 4 is used for prompting to overcome resource limitations.
Hyperparameter Settings Our major hyperparameter settings follow Zhao and Schütze (2021). A fixed learning rate (1e-5) is used for all of our experiments without any learning rate schedule to compare finetuning with prompting (Le Scao and Rush, 2021). We use a smaller batch size of 8 for finetuning and prompting because it achieves slightly better performance. We use the max sequence length of 256. The model is trained for 50 epochs and we select the checkpoint by validation accuracy for testing as suggested in Mosbach et al. (2021); Zhang et al. (2021a). The α value for β distribution in prompt mixup is set to 1.2 for all of the experiments.
Prompting The language sets L used for multilingual verbalizers are determined by the language availability of the dataset. Specifically, for XNLI, L = {EN, AR, BG, DE, EL, ES, FR, HI, RU, SW, TH, TR, UR, VI, ZH}. For PAWS-X, L = {EN, DE, ES, FR, JA, KO, ZH}
For simplicity, the verbalizers of target languages are translated by Google Translate. Similar with XNLI, we use "paraphrase → yes" and "nonparaphrase → no" as the verbalizer of PAWS-X in English. Table 6 presents the full multilingual verbalizer we use for the PAWS-X dataset.
We discuss Universal Prompting across languages for multilingual sentence-pair classification tasks in Section 2. Moreover, we believe the same notion of alleviating source-target discrepancies in terms of prompt template and verbalizer is generally applicable for cross-lingual tasks, which is left for future work.

heading: B Generalizability of Prompting Word Removal
text: In Section 2, we show that by removing template words, UP provides a more reasonable baseline for cross-lingual prompting on XNLI. To see whether such a removal generalizes to other cross-lingual sentence-pair classification task, we also investigate the impact of removing template words on PAWS-X, as shown in Table 7. We find that UP still performs reasonably well on PAWS-X without template words. It was also shown in IV et al. (2021) that hand-engineering prompt is less important when PLMs are finetuned for monolingual tasks. Our UP generalizes this in cross-lingual tasks.

heading: C Performance with Standard Deviation
text: In Table 8 and 9, we show the performance with standard deviation specifically in every language.
M et
ho d
E N
A R
B G
D E
E L
E S
FR H
I R
U SW
T H
T R
U R
V I
Z H
Av g.
16 sh
ot s
FT 3 5 .6 2 2 .4
5 3 5 .1 1 1 .5
3 3 4 .8 5 1 .8
7 3 5 .0 7 2 .1
1 3 5 .0 8 2 .2
2 3 5 .2 1 2 .2
0 3 4 .9 5 2 .0
9 3 4 .8 9 1 .8
4 3 4 .5 2 1 .5
6 3 5 .0 7 2 .0
1 3 4 .9 2 1 .5
0 3 4 .7 9 1 .7
7 3 5 .0 2 1 .7
0 3 5 .0 2 1 .8
9 3 4 .7 1 1 .5
2 3 4 .9 9 1 .8
4
U P
4 7 .6 8 1 .6
5 4 2 .0 1 3 .6
8 4 5 .5 0 2 .3
2 4 4 .5 1 2 .4
3 4 6 .6 8 2 .7
5 3 6 .6 1 4 .1
8 4 6 .8 1 1 .8
5 4 0 .2 9 5 .1
9 4 5 .4 3 2 .1
1 4 2 .0 6 4 .0
0 4 4 .2 1 3 .6
4 4 1 .0 4 4 .1
9 4 0 .6 1 4 .7
6 4 5 .7 9 2 .1
0 3 8 .4 2 5 .5
4 4 3 .1 8 2 .7
7
O U
R S
4 8 .5 5 1 .4
3 46
.2 4 2
.6 1
47 .9
5 1 .7
2 48
.0 0 1
.5 3
4 7 .4 1 2 .2
7 47
.4 7 1
.8 3
48 .6
1 1 .4
8 44
.3 6 2
.4 8
4 6 .7 6 1 .5
5 4 4 .3 5 1 .8
8 4 5 .9 5 2 .4
1 45
.8 3 1
.5 1
44 .8
0 2 .2
0 47
.3 1 1
.6 7
44 .5
5 3 .7
6 46
.5 4 1
.8 3
W /O
M V
49 .5
4 2 .6
2 4 1 .5 5 3 .5
6 4 6 .8 4 2 .0
5 4 5 .5 3 2 .3
9 4 7 .5 9 2 .5
9 3 4 .6 3 1 .1
1 4 8 .5 5 2 .2
1 4 2 .3 9 4 .2
5 47
.1 8 1
.8 7
4 3 .9 5 2 .4
0 46
.3 7 2
.3 1
4 3 .8 2 3 .9
2 4 3 .3 2 3 .6
6 4 6 .5 2 2 .3
1 4 0 .0 9 3 .5
1 4 4 .5 2 2 .1
5
W /O
M IX
U P
4 8 .3 8 1 .8
0 4 5 .5 9 1 .9
6 4 7 .7 4 1 .9
4 4 7 .7 2 2 .0
7 47
.6 0 2
.5 7
4 4 .3 8 5 .7
8 4 7 .8 3 1 .5
4 4 2 .4 4 3 .3
1 4 6 .6 9 1 .3
1 44
.3 8 1
.3 9
4 4 .6 5 2 .5
3 4 5 .5 2 2 .0
2 4 3 .4 8 2 .6
6 4 6 .6 5 1 .6
7 4 0 .8 3 4 .3
1 4 5 .5 9 1 .9
1
32 sh
ot s
FT 3 7 .6 2 2 .6
6 3 6 .8 2 2 .0
9 3 6 .6 1 2 .2
8 3 7 .0 3 2 .5
6 3 7 .0 7 1 .9
0 3 7 .3 9 2 .1
7 3 7 .5 3 1 .7
1 3 7 .3 5 1 .5
5 3 6 .8 3 2 .3
7 3 6 .4 2 1 .9
3 3 6 .4 0 1 .7
2 3 6 .4 0 1 .5
9 3 6 .7 1 1 .5
0 3 6 .8 4 2 .3
2 3 6 .9 6 2 .1
7 3 6 .9 3 1 .9
6
U P
5 3 .3 3 1 .6
0 4 7 .7 0 3 .5
8 5 0 .8 7 1 .4
1 4 9 .7 4 2 .6
6 5 1 .4 1 1 .9
8 4 1 .4 8 4 .7
7 5 1 .0 9 1 .4
9 4 4 .9 7 1 .4
0 5 0 .1 1 0 .7
6 4 6 .7 6 3 .0
0 4 9 .5 0 1 .3
6 4 5 .9 2 2 .5
3 4 5 .6 4 2 .0
7 5 1 .0 0 1 .5
4 4 4 .3 3 2 .5
1 4 8 .2 6 1 .3
4
O U
R S
5 2 .7 9 2 .1
4 49
.3 7 2
.8 5
51 .4
8 1 .7
9 50
.8 4 1
.9 8
51 .7
8 2 .0
7 5 0 .0 5 3 .6
0 51
.7 7 2
.1 3
48 .0
8 1 .8
4 50
.4 6 1
.8 9
4 7 .3 0 2 .8
9 4 9 .3 5 2 .3
0 50
.1 4 1
.7 7
47 .4
4 2 .3
0 50
.8 4 2
.5 0
48 .2
5 4 .0
8 49
.9 9 2
.2 1
W /O
M V
53 .7
5 1 .5
3 4 8 .4 2 1 .0
6 5 0 .7 1 1 .6
7 5 0 .5 7 1 .2
1 5 1 .7 6 1 .7
6 4 1 .9 8 4 .8
5 5 1 .5 4 1 .7
8 4 5 .6 4 3 .7
6 50
.4 6 1
.1 4
4 5 .8 4 3 .3
2 49
.6 5 0
.9 1
4 7 .4 2 3 .5
7 4 5 .5 8 2 .5
7 5 0 .5 6 1 .6
3 4 7 .5 4 2 .3
0 4 8 .7 6 1 .5
6
W /O
M IX
U P
5 2 .3 8 2 .5
0 4 9 .2 9 2 .0
5 5 1 .3 9 1 .7
1 5 0 .7 6 1 .7
5 5 1 .6 0 1 .4
4 50
.2 1 3
.0 0
5 1 .5 4 1 .8
3 4 7 .5 7 0 .5
1 5 0 .3 5 1 .6
6 47
.5 6 1
.0 2
4 9 .0 7 1 .7
6 4 9 .5 6 1 .2
5 4 7 .0 2 1 .2
3 5 0 .6 5 1 .4
5 4 6 .2 4 2 .0
7 4 9 .6 8 1 .4
6
64 sh
ot s
FT 4 2 .9 7 3 .8
8 4 0 .7 0 4 .0
6 4 1 .2 9 3 .9
3 4 1 .6 8 3 .6
6 4 2 .0 9 3 .8
1 4 2 .4 6 4 .0
9 4 2 .2 3 4 .3
8 4 0 .5 9 3 .3
9 4 0 .3 8 3 .5
8 3 9 .9 6 3 .1
0 4 0 .6 5 3 .4
8 4 0 .8 4 3 .2
6 4 0 .2 4 3 .0
1 4 2 .0 9 3 .6
8 4 0 .5 3 3 .4
6 4 1 .2 5 3 .6
0
U P
5 7 .7 6 1 .4
9 5 1 .6 7 2 .3
1 5 4 .8 5 1 .5
2 5 4 .9 9 1 .7
1 5 4 .6 9 1 .0
0 5 1 .6 3 3 .9
8 5 4 .9 6 0 .8
7 4 7 .9 7 1 .7
4 5 3 .3 2 1 .7
3 4 8 .1 2 1 .2
1 5 1 .9 1 1 .2
7 4 9 .8 9 2 .8
0 4 7 .8 6 2 .1
0 5 4 .1 4 1 .2
1 4 9 .1 3 3 .3
8 5 2 .1 9 1 .5
4
O U
R S
59 .9
7 1 .2
5 5 3 .1 8 1 .1
2 5 6 .5 1 1 .1
4 56
.6 7 1
.1 2
5 5 .6 3 0 .6
8 56
.7 9 1
.1 7
56 .9
7 1 .5
2 5 1 .7 7 1 .0
9 5 5 .4 6 1 .2
6 5 0 .7 1 1 .5
0 5 3 .3 5 1 .3
6 54
.2 1 1
.0 6
5 0 .7 6 1 .3
9 5 6 .0 5 0 .9
7 5 3 .0 9 1 .2
0 5 4 .7 4 0 .9
3
W /O
M V
5 9 .1 7 1 .5
9 53
.7 9 1
.6 6
56 .9
5 0 .9
0 5 6 .5 3 1 .2
3 56
.1 8 0
.9 9
5 5 .3 5 3 .3
9 5 6 .4 8 1 .9
6 52
.1 7 1
.2 2
55 .7
2 1 .1
8 50
.8 9 1
.3 6
54 .5
5 0 .9
1 5 3 .3 5 1 .2
7 51
.6 2 1
.0 5
56 .4
3 1 .1
4 54
.4 2 1
.0 4
54 .9
1 1 .1
8
W /O
M IX
U P
5 9 .5 6 1 .2
2 5 3 .0 6 1 .0
2 5 5 .9 8 1 .0
5 5 5 .6 5 1 .0
1 5 5 .1 6 0 .4
8 5 6 .6 7 1 .1
0 5 6 .6 6 1 .3
0 5 1 .4 4 1 .4
5 5 5 .1 8 1 .1
8 4 9 .9 9 1 .6
0 5 2 .9 0 0 .8
5 5 3 .7 6 0 .9
2 4 9 .8 0 1 .8
9 5 5 .4 3 0 .8
7 5 3 .7 0 1 .4
5 5 4 .3 3 0 .9
8
12 8s
ho ts
FT 4 7 .2 4 4 .5
0 4 3 .9 1 2 .4
3 4 4 .1 3 2 .6
3 4 3 .9 6 2 .8
3 4 4 .3 8 2 .1
3 4 5 .2 5 3 .3
8 4 4 .4 8 2 .8
9 4 2 .3 8 2 .6
8 4 2 .8 1 2 .4
2 4 2 .8 7 1 .7
7 4 2 .8 7 2 .4
8 4 2 .9 3 2 .5
3 4 2 .3 6 2 .3
7 4 4 .6 0 2 .7
4 4 2 .8 7 2 .5
7 4 3 .8 0 2 .5
8
U P
6 0 .0 8 2 .5
6 5 1 .3 1 6 .0
2 5 6 .6 0 3 .3
0 5 5 .1 0 3 .8
4 5 6 .1 7 1 .8
1 5 1 .2 5 1 0 .3
1 5 6 .9 7 2 .1
9 4 9 .6 2 4 .3
3 5 5 .1 8 1 .7
5 4 8 .7 1 3 .9
4 5 3 .8 7 2 .0
5 5 0 .4 2 3 .7
9 4 9 .2 0 3 .3
9 5 5 .0 3 4 .0
4 5 3 .1 5 2 .6
1 5 3 .5 1 3 .5
1
O U
R S
62 .5
7 1 .6
9 5 4 .9 1 1 .9
3 5 8 .7 2 1 .3
1 58
.8 1 1
.4 9
58 .2
5 0 .9
9 59
.4 7 1
.7 9
5 8 .7 6 1 .2
6 52
.9 3 1
.6 5
57 .3
5 1 .3
8 5 0 .9 5 1 .3
4 5 4 .3 0 2 .3
9 54
.9 4 1
.7 6
5 1 .4 7 2 .3
7 5 7 .8 0 1 .5
0 5 4 .9 9 1 .8
6 56
.4 2 1
.3 7
W /O
M V
6 1 .5 1 1 .5
8 55
.3 1 1
.2 6
5 8 .6 7 1 .3
8 5 8 .1 5 1 .8
3 5 8 .1 2 1 .0
2 5 8 .1 0 1 .7
4 5 8 .4 2 0 .5
0 5 2 .3 1 1 .0
3 5 6 .9 9 1 .0
9 5 0 .8 0 1 .3
0 55
.4 0 0
.9 7
5 3 .8 8 0 .8
1 51
.7 4 0
.7 6
57 .9
6 1 .3
2 56
.1 2 1
.5 0
5 6 .2 3 0 .9
0
W /O
M IX
U P
6 1 .8 4 1 .6
4 5 4 .5 9 1 .4
1 58
.7 7 1
.5 5
5 8 .5 7 1 .2
4 5 7 .7 7 1 .7
4 5 9 .1 3 1 .9
2 58
.8 9 1
.3 4
5 2 .7 0 1 .8
6 5 6 .9 9 1 .8
3 52
.0 5 1
.7 1
5 4 .1 5 1 .5
8 5 4 .6 9 1 .3
8 5 1 .3 1 1 .0
7 5 7 .2 7 1 .9
2 5 5 .5 9 2 .4
4 5 6 .2 9 1 .4
6
25 6s
ho ts
FT 5 9 .4 9 2 .1
1 5 2 .8 7 2 .3
4 5 5 .9 2 2 .1
5 5 5 .5 1 2 .2
5 5 5 .0 7 2 .6
0 5 7 .4 4 1 .7
4 5 6 .3 2 2 .0
4 5 1 .7 5 2 .7
2 5 4 .1 9 2 .4
6 4 9 .8 8 2 .5
1 5 2 .3 8 2 .9
6 5 3 .6 8 1 .8
0 5 0 .3 8 2 .3
4 5 5 .3 7 2 .0
2 5 3 .9 5 2 .0
8 5 4 .2 8 2 .1
5
U P
6 5 .0 8 1 .4
9 5 6 .5 7 2 .0
8 6 1 .0 3 1 .8
6 6 0 .6 5 1 .2
7 6 0 .7 4 1 .8
4 5 9 .2 1 2 .6
4 6 1 .0 1 2 .4
1 5 5 .1 8 2 .5
4 5 9 .4 1 2 .2
9 5 3 .7 3 2 .2
7 5 7 .6 6 2 .0
8 5 7 .6 2 2 .6
5 5 4 .0 8 2 .4
7 6 0 .5 8 1 .8
4 5 8 .7 1 2 .1
6 5 8 .7 5 1 .9
2
O U
R S
67 .9
7 1 .0
2 59
.5 4 1
.3 6
63 .5
9 1 .2
3 63
.2 6 1
.2 3
62 .3
4 0 .9
1 64
.8 0 1
.1 7
63 .9
3 1 .0
6 58
.3 9 0
.7 1
61 .8
7 1 .5
5 55
.8 3 1
.5 2
59 .1
9 1 .0
6 60
.3 2 1
.0 8
56 .0
0 0 .9
5 62
.4 1 0
.5 2
61 .2
9 1 .1
4 61
.3 8 0
.9 2
W /O
M V
6 5 .8 0 1 .2
5 5 8 .0 7 1 .2
2 6 2 .0 4 1 .6
0 6 1 .3 3 1 .4
5 6 1 .0 5 1 .2
4 6 3 .0 3 1 .4
3 6 2 .3 6 1 .2
2 5 6 .1 6 0 .5
2 6 0 .1 4 1 .5
3 5 4 .1 7 1 .0
0 5 8 .2 3 1 .2
4 5 7 .6 2 1 .0
2 5 4 .1 2 0 .9
5 6 0 .5 2 1 .3
4 5 9 .8 1 1 .2
4 5 9 .6 3 0 .9
2
W /O
M IX
U P
6 7 .4 0 0 .6
5 5 8 .0 2 1 .4
1 6 2 .3 3 1 .4
7 6 2 .1 8 1 .0
0 6 1 .3 5 1 .2
4 6 3 .6 1 1 .2
4 6 2 .9 3 1 .2
5 5 6 .8 9 1 .0
6 6 0 .7 5 1 .6
7 5 4 .6 8 1 .6
0 5 8 .0 6 1 .5
1 5 9 .0 0 1 .4
4 5 4 .7 4 1 .1
0 6 1 .1 7 1 .1
9 5 9 .3 3 1 .0
8 6 0 .1 6 0 .9
7
Ta bl
e 8:
Z er
osh
ot cr
os s-
lin gu
al tr
an sf
er ac
cu ra
cy w
ith st
an da
rd de
vi at
io n
on X
N L
I. FT
:fi ne
tu ni
ng ;U
P: U
ni ve
rs al
Pr om
pt in
g; M
V :m
ul til
in gu
al ve
rb al
iz er
.R ep
or te
d re
su lts
ar e
av er
ag ed
w ith
5 ra
nd om
se ed
s.
M et
ho d
E N
D E
E S
FR JA
K O
Z H
Av g.
25 6s
ho ts
FT 6 3 .1 8 5 .6
9 6 0 .8 1 5 .0
4 6 0 .9 5 3 .8
3 6 1 .3 9 4 .3
8 5 8 .6 0 3 .8
5 5 8 .4 8 2 .5
9 5 9 .7 8 4 .7
1 6 0 .4 6 4 .2
3
U P
6 5 .5 0 3 .9
8 6 2 .2 1 3 .9
0 6 3 .2 4 3 .5
2 6 2 .8 2 4 .3
1 5 4 .1 1 5 .3
7 5 4 .3 0 4 .9
8 5 5 .9 9 4 .6
4 5 9 .7 4 4 .1
2
O U
R S
71 .8
7 5 .0
3 68
.5 9 4
.5 0
69 .1
0 4 .9
2 69
.0 2 3
.9 7
60 .4
1 2 .6
9 60
.8 8 1
.7 6
62 .7
5 3 .4
6 66
.0 9 3
.6 2
W /O
M V
6 9 .0 6 4 .5
8 6 6 .2 6 4 .1
3 6 6 .4 7 4 .0
5 6 5 .7 9 4 .1
3 5 9 .2 8 5 .3
6 5 8 .3 4 4 .2
4 6 0 .7 7 6 .0
0 6 3 .7 1 4 .3
7
W /O
M IX
U P
7 0 .9 5 4 .1
7 6 7 .1 4 3 .5
3 6 7 .5 8 4 .3
6 6 7 .6 3 4 .0
6 5 9 .0 1 1 .6
7 6 0 .4 4 1 .5
4 6 1 .1 6 2 .4
1 6 4 .8 4 2 .9
1
51 2s
ho ts
FT 7 7 .6 4 8 .3
8 7 3 .4 1 6 .3
0 7 3 .1 9 6 .8
4 7 4 .3 3 6 .4
7 6 5 .5 5 3 .8
9 6 5 .1 9 3 .8
1 6 8 .2 5 5 .2
8 7 1 .0 8 5 .8
1
U P
8 3 .3 1 2 .4
3 7 6 .1 8 1 .9
3 7 7 .6 3 1 .7
3 7 7 .4 2 1 .6
4 6 3 .4 1 2 .3
9 6 5 .0 3 2 .1
2 6 8 .0 6 0 .7
8 7 3 .0 1 1 .5
2
O U
R S
84 .9
7 1 .6
0 78
.6 3 1
.3 6
7 9 .6 0 1 .3
1 80
.4 8 1
.0 9
67 .8
6 1 .2
7 6 8 .1 3 0 .8
8 72
.3 4 1
.4 7
76 .0
0 1 .0
4
W /O
M V
8 4 .8 1 1 .1
9 7 8 .5 6 0 .7
8 79
.6 7 0
.8 1
7 9 .6 4 0 .3
6 6 7 .0 4 1 .8
5 68
.3 4 0
.8 8
7 1 .5 0 1 .7
6 7 5 .6 5 0 .6
4
W /O
M IX
U P
8 4 .8 4 1 .5
7 7 7 .8 5 1 .9
4 7 9 .3 6 2 .0
1 7 9 .6 9 1 .9
9 6 6 .7 6 3 .2
4 6 8 .0 3 2 .1
9 7 1 .0 3 2 .6
5 7 5 .3 7 2 .0
0
Ta bl
e 9:
Ze ro
-s ho
tc ro
ss -li
ng ua
lt ra
ns fe
ra cc
ur ac
y w
ith st
an da
rd de
vi at
io n
on PA
W S-
X .F
T: fin
et un
in g;
U P:
U ni
ve rs
al Pr
om pt
in g;
M V
:m ul
til in
gu al
ve rb
al iz
er .R
ep or
te d
re su
lts ar e av er ag ed w ith 5 ra nd om se ed s.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section 6
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 4

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 4

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Section 4

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 4
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section 4, Appendix A
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 4
C 3 Did you run computational experiments? Section 4, Appendix B, Appendix C

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix A
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 4

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix A
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Moreover, we believe the same notion of alleviating source-target discrepancies in terms of prompt template and verbalizer is generally applicable for cross-lingual tasks, which is left for future work