Dual-Alignment Pre-training for Cross-lingual Sentence Embedding
Year: 2023

Abstract:
Recent studies have shown that dual encoder models trained with the sentence-level translation ranking task are effective methods for cross-lingual sentence embedding. However, our research indicates that token-level alignment is also crucial in multilingual scenarios, which has not been fully explored previously. Based on our findings, we propose a dual-alignment pre-training (DAP) framework for cross-lingual sentence embedding that incorporates both sentence-level and token-level alignment. To achieve this, we introduce a novel representation translation learning (RTL) task, where the model learns to use one-side contextualized token representation to reconstruct its translation counterpart. This reconstruction objective encourages the model to embed translation information into the token representation. Compared to other token-level alignment methods such as translation language modeling, RTL is more suitable for dual encoder architectures and is computationally efficient. Extensive experiments on three sentencelevel cross-lingual benchmarks demonstrate that our approach can significantly improve sentence embedding. Our code is available at https://github.com/ChillingDream/DAP.

heading: 1 Introduction
text: Cross-lingual sentence embedding encodes multilingual texts into a single unified vector space for a variety of Natural Language Processing (NLP) tasks, including cross-lingual sentence retrieval (Artetxe and Schwenk, 2019b) and crosslingual natural language inference (Conneau et al., 2018). The text sequences can be efficiently retrieved and compared using the inner product between their dense representations.
The task of sentence embedding now heavily depends on pre-trained language models (Devlin
∗Work done during internship at Microsoft. †Corresponding Author.
et al., 2019; Conneau and Lample, 2019; Conneau et al., 2020b,a). By fine-tuning the CLS token of the pre-trained model, they encode the input text sequence into a single vector representation. Recent research has shown that using the translation ranking task in combination with a dual pretrained encoder can result in superior sentence embeddings (Yang et al., 2019; Chidambaram et al., 2019; Yang et al., 2021; Chi et al., 2021; Feng et al., 2022). The purpose of fine-tuning the CLS token is to learn sentence-level alignment and to compress the entire sentence’s information into the CLS token. This method makes the CLS tokens of semantically relevant sentences have larger inner products. However, token-level alignment in multilingual scenarios is also crucial, and the fine-grained alignment task in cross-lingual sentence embedding has not been fully explored. As shown in Figure 1, we visualize the token representation similarities between a pair of parallel corpora. Training for an objective solely with regard to CLS token causes the token representations to disperse across the embedding space.
3466
Based on our observations, we propose an efficient dual-alignment pre-training (DAP) framework for cross-lingual sentence embedding. The embedding model is trained towards both sentencelevel alignment and token-level alignment. Previous cross-lingual pre-training studies (Chi et al., 2021; Feng et al., 2022) employ translation language modeling (TLM) to achieve token alignment. In this paper, we introduce a novel representation translation learning (RTL) method that reconstructs the entire English input based on the token representations of parallel non-English sentences using a transformer model. By optimizing the RTL objective, the model learns to embed the information of English sentences into the representation of its non-English counterpart. Unlike TLM, computing RTL only needs one-side self-contextualized representation and does not involve extra feedforward propagation. We train our model on public corpora and evaluate it on three cross-lingual tasks: bitext retrieval, bitext mining, and cross-lingual natural language inference. Our results demonstrate DAP can effectively improve cross-lingual sentence embedding.
Our contributions are summarized as follows:
• We propose a novel cross-lingual pre-training framework DAP for sentence-level tasks, achieving both sentence-level and token-level alignment by representation translation learning, which is more suitable for dual encoders and computationally efficient compared with previous alignment methods.
• Extensive experiments on three cross-lingual tasks demonstrate DAP significantly improves sentence embedding.
• We train a model on a moderate-size dataset and find its performance comparable with that of the large-scale state-of-the-art pre-trained model.

heading: 2 Related Work

heading: 2.1 Cross-lingual Pre-training
text: Following the success of BERT for English (Devlin et al., 2019), multilingual BERT comes out by building a shared multilingual vocabulary and training on multiple monolingual corpora with the masked language modeling (MLM) objective. XLM (Conneau and Lample, 2019) proposes a translation language modeling (TLM) task which is
the extension of MLM to bitext corpora, so that the model can learn the cross-lingual alignment from translation pairs. Unicoder (Huang et al., 2019) introduces three bitext pre-training tasks to help the model capture cross-lingual information from more perspectives. XLM-R (Conneau et al., 2020a) scales up the amount of monolingual data and training time. They achieve better performance than previous works without using parallel corpora.

heading: 2.2 Sentence Embedding
text: The dual encoder architecture is first proposed by Guo et al. (2018). They encode the source and target sentences to a unified embedding space, respectively, and compute the similarity score using inner product. The model is trained under a translation ranking task to make the model score higher for translation pairs than the negative examples. Yang et al. (2019) enhances the dual encoder by additive margin softmax, which further enlarges the distance between negative pairs. Based on additive margin softmax, LaBSE (Feng et al., 2022) combines the translation ranking task with MLM task and TLM task and trains on a larger corpus. InfoXLM (Chi et al., 2021) interprets the MLM, TLM and translation ranking task used in cross-lingual pre-training in a unified informationtheoretic framework, based on which they propose cross-lingual contrastive learning to maximize sentence-level mutual information.

heading: 3 Method

heading: 3.1 Preliminaries
text: Transformer Encoder Transformer encoder has been widely adopted in modern language models (Vaswani et al., 2017; Devlin et al., 2019; Conneau and Lample, 2019). It consists of an embedding layer and L stacked transformer blocks with self-attention modules. Each input token xi will be encoded into a vector space as the initial hidden vector h0i . Then, in each transformer block, the hidden vector of the i-th token hli is computed from the self-attentive fusion of all hidden vectors output from the previous layer:
hl = (hl1, h l 2, · · · , hlS) = f l(hl−1). (1)
We finally get the contextualized token representation f(x) = fL(fL−1(· · · f1(h0))).
Cross-lingual Pre-training Masked language modeling (MLM) (Devlin et al., 2019) and Transla-
tion language modeling (TLM) (Conneau and Lample, 2019) are two typical tasks for cross-lingual pre-training. MLM is conducted on monolingual corpora. A randomly selected subset of input tokens will be replaced by a special [MASK] token or another random token, and models learn to recover these corrupted tokens according to the context. TLM extends MLM to cross-lingual scenarios with the following objective:
LTLM (x, y) = ℓ (x⊕ y, f(m(x)⊕m(y))) , (2)
where ⊕ denotes sequence concatenation operator and m denotes element-wise random replacement. During training, models can predict the masked token using the unmasked token in the translation. In this way, models learn cross-lingual token-level alignment using the parallel corpora.
However, TLM is designed for a cross-encoder architecture in which tokens from the source and target sentences are mutually accessible in intermediate layers. As a result, models trained with TLM may rely on this information exchange, which is not available during the inference stage when sentences are independently encoded. Additionally, computing TLM requires an extra feedforward propagation, which inputs concatenated sentence pairs, resulting in increased training costs. Our proposed representation translation learning task can overcome both the weaknesses.

heading: 3.2 Model Structure
text: Our dual-alignment pre-training framework contains two transformer models: dual encoder model
f and representation translation learning (RTL) head g.
For the encoder model, we adopt the most popular BERT architecture with 12 layers of transformer encoder blocks, 12 attention heads, and 768-dimension hidden states. Following Devlin et al. (2019), we prepend a special token [CLS] to the input:
f(x) = f([CLS], x1, . . . , xS). (3)
We take the hidden vector of CLS token hLcls as the representation of the whole sentence fs(x). Like other multilingual language models, our model is language-agnostic, which means all languages share the same single transformer.
The RTL head is a stack of K transformer encoder blocks with a vocabulary prediction head at the top. The function of RTL head is to reconstruct the translation sentence y from the token representations of the source sentence hL (source sentences indicate non-English sentences in this paper):
g(h, y) = π ( W T gK ( gK−1 ( · · · g0(h, y) ))) ,
g0(h, y)=(hL1,· · · ,hLSx ,[MASK],· · ·,[MASK]︸ ︷︷ ︸ ×Sy ),
(4)
where π is softmax function and W is the weight matrix of the vocabulary prediction head. In our experiments, we find a small RTL head with K = 2 performs best generally.

heading: 3.3 Pre-training Tasks
text: To achieve both sentence-level and token-level alignment, we design a pre-training framework consisting of two tasks: translation ranking task and representation translation learning task. These two objectives are leveraged simultaneously during training. The whole procedure is depicted in Figure 2.

heading: 3.3.1 Translation Ranking
text: Dual encoder models trained with the translation ranking (TR) task have been proven effective in learning cross-lingual embeddings (Yang et al., 2019; Feng et al., 2022; Chi et al., 2021). These models learn to maximize the similarity of the embedding pairs of parallel sentences and the dissimilarity of mismatched pairs. Therefore, they are well suited for solving retrieval and mining tasks that use inner product as ranking metrics. Following (Feng et al., 2022), we formulate the training task as follows:
LTR = − 1
N
N∑
i=1
log eϕ(xi,yi)
∑B j=1 e ϕ(xi,yj) , (5)
where B is the batch size and ϕ(x, y) is defined as the similarity of the representation of each text, typically fs(x)T fs(y). In this paper, we use the hidden vector of CLS token to represent the sentence.

heading: 3.3.2 Representation Translation Learning
text: Minimizing LTR essentially maximize the lower bound of the mutual information I(x; y) (Oord et al., 2018; Chi et al., 2021). However, it is hard for models to find an embedding perfectly containing all information of the sentence. Consequently, models may only pay attention to the high-level global information and neglect some local tokenlevel information. To this end, we add an auxiliary loss to force the models to preserve the token-level information throughout the entire model:
LRTL = 1
S
S∑
i=1
CE(g(f∗(x), y)i, yi), (6)
where f∗(x) denotes all hidden vectors of x except CLS and CE denotes cross entropy. It is worth noting that we do not involve the CLS token in calculating RTL objective because we find it will make translation ranking objective hard to converge. To train the RTL head with a stable and consistent
target, the reconstruction direction is always from non-English sentences to their English translations.
Combining with the translation ranking objective we get the final loss:
LDAP = LTR + LRTL. (7)
As RTL does not need an extra feedforward propagation, RTL only introduces a little computation and will not slow down the pre-training significantly. The only time-consuming operation is the softmax over the huge vocabulary which can be further relieved by techniques like negative sampling and hierarchical softmax (not used in our experiments).

heading: 4 Experiments
text: In this section, we first describe the training setup. Then we compare our method with previous works on three sentence-level cross-lingual tasks.

heading: 4.1 Pre-training data
text: Following Artetxe and Schwenk (2019b) we collect parallel training data for 36 languages (used in XTREME Tatoeba benchmark) by combining Europarl, United Nations Parallel Corpus, OpenSubtitles, Tanzil, CCMatrix and WikiMatrix corpora, which are downloaded from OPUS website (Tiedemann, 2012). As stated in section 3.3, we align all other languages with English, so we only collect parallel corpora that contain English. For each non-English language, we retain at most 1 million sentence pairs at random. The whole dataset has 5.7GB data, which is far less than typical largescale pre-training (Feng et al., 2022; Chi et al., 2021), but our method still achieves performance comparable with the state-of-the-art.

heading: 4.2 Implementation Details
text: We initialize the encoder model from multilingual BERT base or XLM-R base, respectively, using the checkpoint published on Huggingface model hub, and initialize the K-layer RTL head from the last K transformer layers by the corresponding encoder model. The maximum sentence length is restricted to 32 tokens, and sentences longer than 32 tokens will be truncated. We train the model for 100,000 steps using the AdamW optimizer with a learning rate of 5e-5 and a total batch size of 1024 on 8 Tesla V100 GPUs for 1 day. The results reported are the average of three different seeds.

heading: 4.3 Compared models
text: To demonstrate the effectiveness of our proposed Representation Translation Learning, we first compare it with the base models (mBERT or XLM-R) and their TR-finetuned versions. Additionally, we also introduce a variant of our method that leverages TLM.
Furthermore, we also compare our approach with two state-of-the-art multilingual language models, InfoXLM (Chi et al., 2021) and LaBSE (Feng et al., 2022). It is worth noting that InfoXLM and LaBSE use 10 times more training data than our method and are trained longer with a larger batch size.

heading: 4.4 Bitext Retrieval
text: In bitext retrieval, given a query sentence from source language, models need to retrieve the most relevant sentence among a collection of sentences in the target language. Following previous works (Feng et al., 2022; Chi et al., 2021; Artetxe and Schwenk, 2019b), we use the Tatoeba dataset to evaluate our pre-training framework in a zeroshot manner.
Tatoeba contains parallel sentences in more than 300 languages, and we use the 36 languages version from XTREME benchmark (Hu et al., 2020). Each language has up to 1000 sentences paired with English.
Results We test on all 36 languages and report the average accuracy over 14 languages tested in LASER (Artetxe and Schwenk, 2019b) and 36 languages tested in XTREME. Besides, we set up a new group of 28 languages based on our observation of the low-resource test languages. Among the original 36 languages, some scarce languages have less than 1000 sentence pairs, and some of them even only have about 200 sentence pairs, and we observe that the accuracy of these languages is inconsistent between the two retrieval directions ("en→xx" and "xx→en" with a difference more than 30%) and also significantly lower than other languages with abundant resources. This indicates that the results obtained from small test sets are not as reliable as those from larger test sets. Therefore, we report a 28-language version where all languages contain 1000 test pairs. The retrieval accuracy for each language is reported in the appendix A.
In Table 1, we observe that our DAP method outperforms all other variants significantly. mBERT and XLM-R perform the worst because they lack a sentence-level objective. TLM improves TR’s performance in the direction "en→xx" but hurts direction "xx→en". By contrast, DAP brings consistent improvement. Compared with the two state-of-theart methods, our method performs much better than InfoXLM and only slightly falls behind LaBSE.
Considering the training cost, we think this result has demonstrated DAP’s potential.

heading: 4.5 Bitext Mining
text: In bitext mining, models need to detect the parallel sentence pairs (e.g., translations) from a pair of monolingual corpus. We use the BUCC 2018 dataset (Zweigenbaum et al., 2017) to perform evaluations, which contains four language pairs: fr-en, de-en, ru-en and zh-en. Each corpus contains 150k to 1.2M unpaired sentences and gold labels telling which sentences are translation pairs.
Following Artetxe and Schwenk (2019a), we employ the ratio between the cosine of a given candidate and the average cosine of its neighbours in both directions. The training set is used to learn the best threshold (Schwenk, 2018) to decide which pairs should be selected. More details of the scoring function and threshold can be found
in appendix B.
Results Table 2 shows the precision, recall and F1 score for four language pairs on training set after optimization. The results of LaBSE are produced using the checkpoints publicized in Huggingface model hub. We do not report the results of InfoXLM because this task was not evaluated in the original paper and we failed to produce reasonable results.
Our method outperforms all variants and even LaBSE, which means our model learns an embedding space with better separability. When testing the optimized model on test set, our model shows remarkable generalization ability and enlarges the gap against other methods as shown in Table 3. We outperform the state-of-the-art LaBSE by 0.9% and other variants by at least 3.0%. Similar to the retrieval task, mBERT and XLM-R perform the
worst. TLM brings improvements for zh-en but gets worse for fr-en. DAP consistently performs the best on all metrics. Furthermore, the improvement observed in DAP’s performance is larger in comparison to the retrieval task. This indicates that DAP is more effective in enhancing performance on complex tasks, suggesting its potential as a valuable tool for addressing challenging problems.

heading: 4.6 Cross-lingual Natural Language Inference
text: Natural language inference (NLI) is a well-known task to evaluate models’ classification performance under fine-tuning. The goal is to predict the relationship between the input sentence pair. The candidate relationships are entailment, contradiction and neutral. XNLI (Conneau et al., 2018) extends NLI to the multilingual setting of 15 languages. Following Chi et al. (2021), we fine-tune the model with the English training set and directly evaluate on test sets of other languages. The hyperparameters of fine-tuning are reported in the appendix C.
Results Table 4 shows accuracy for 15 languages. We observe that the differences between variants are relatively small compared with retrieval and mining tasks. We think this is because judging the relationship between two sentences does not rely on cosine similarity, so the pre-training cannot be directly transferred to the downstream task. mBERT variants all show positive results and DAP has the largest improvement. But for XLM-R variants, only DAP maintains the performance as the base model. The TR and TLM variants suffer from performance degradation. We think this is because XLM-R has already been a well-trained multilingual model and our continued pre-training
is insufficient to improve the classification capacity. However, we demonstrate DAP will not harm classification performance for a well-trained base model.

heading: 5 Analysis
text: In this section, we conduct experiments to get a deeper understanding of DAP. In each setting, we report the average accuracy over 36 languages and two retrieval directions on Tatoeba, average F1 score on BUCC test set and average accuracy on XNLI. All variants are trained from mBERT.

heading: 5.1 Translation Direction
text: In our method, the RTL head only learns to translate from non-English to English. Here we investigate if the opposite direction can help the pretraining. To remind the model of the language to be reconstructed, we add language embeddings to the representation before the RTL head like TLM.
As shown in Table 5, translating from English to non-English performs much worse than the opposite direction. Also, the mixed-up training gets an intermediate performance. We attribute the differ-
ence between the two directions to the dispersion of the objective. We assume that RTL aligns the source language’s representation towards the target language. So, if the reconstruction target keeps switching among different languages, it will make RTL hard to converge.

heading: 5.2 Reconstruction Ratio
text: To better understand the objective of the RTL task, we conduct experiments where RTL head only needs to reconstruct partial target sentences with the other target token representations accessible. The tokens to reconstruct are selected randomly with probability ρ. Larger ρ will make the RTL task harder.
From Figure 3, we can find the variants with ρ < 1 have similar performance on all tasks and there is a steep increase at ρ = 1. We think this is because the unmasked target token representations cause information leakage, so the RTL head does not need to learn the alignment from source sentences.

heading: 5.3 Complexity of RTL head
text: We investigate the relation between the RTL head’s complexity and the pre-training performance. We set K = 1, 2, 3, 4 to give RTL head different capabilities to extract aligned information from the representation of the source sentence.
In Figure 4, the three tasks show different tendencies with regard to RTL head’s complexity. Only the accuracy on Tatoeba keeps increasing along with K but the gain from larger K is declining especially after K = 2. For the other two tasks, larger K brings a negative effect. We hypothesize that a smaller K that makes RTL task harder
will enforce the model to generate more informative representations. Setting K = 2 achieves the best general cross-lingual performance across three tasks.

heading: 5.4 Computational Efficiency
text: Computational efficiency is an important factor when designing pre-training tasks. A more efficient method enables models to train on a larger dataset for more steps. We calculate the feedforward floating point operations (FLOPs) for our method and TLM, respectively. In addition, we report the training latency in our training environment. We measure the latency with a total batch size of 512 on 8 Tesla V100 GPUs using PyTorch distributed data parallel.
From Table 6, we can find DAP only increases the training cost by about 50% against the TR-only baseline, which can be further improved if we use negative sampling to reduce the softmax over the huge vocabulary. By contrast, TLM introduces a training cost of more than 150% due to the extra feedforward propagation through the 12-layer encoder. Therefore, DAP is more efficient and scalable for cross-lingual pre-training.

heading: 6 Conclusion
text: In this paper, we find that token-level alignment is crucial for cross-lingual tasks. Based on this observation, we present a dual-alignment pre-training framework for cross-lingual sentence embedding that enables both sentence-level and token-level alignment. The framework consists of a translation ranking task and a newly proposed representation translation learning task, which encourages the token representation to contain all information from its translation counterpart in an efficient way.
We train our models on a moderate-size corpus. The model trained with DAP significantly outperforms variants without token-level alignment or using TLM as the alignment task across three sentence-level cross-lingual tasks, and achieves performance comparable with those state-of-the-art pre-training work trained on 10 times more data with larger batch size and training steps. These results show our approach brings essential improvement for cross-lingual sentence embedding.
Limitations
Although our method is efficient and scalable, we have not conducted pre-training on large-scale corpora due to limited computational resources. The quality and quantity of data are crucial factors for a pre-training model. As our model only covers 36 languages, it cannot provide services for many rare languages. This paper just proposes a new pretraining direction and does not use many training tricks. Exploring DAP’s full capability is left for future work.
Besides, RTL task is not the only possible tokenalignment task for our DAP framework. Other objectives based on token representations are also worth investigating. The best objective form is still under research.

heading: A Full Tatoeba Results
text: We report the Tatoeba retrieval accuracy of all 36 languages in Table 7 and Table 8. Our approach consistently outperforms other baselines in both directions for most languages, with the advantage being particularly significant in the "en→xx" direction. We observed that the performance of the TR-only model can vary much between the two directions, as demonstrated by languages such as jv, kk, sw, and tl. In contrast, our approach exhibits much more stable performance, which is beneficial for bidirectional applications.

heading: B Scoring Function For BUCC
text: In contrast to direction comparison between similarities, margin-based method accounts for the scale inconsistencies of measure. We adopted the method proposed by Artetxe and Schwenk (2019a):
f(x, y) = ϕ(x, y)
∑ z∈Nk(x) ϕ(x,y) k + ∑ z∈Nk(y) ϕ(z,y) k ,
(8) where Nk(x) denotes the set of k nearest neighbours of x in the other language. In our experiments, we set k = 4.
With a certain threshold γ, sentence pairs such that f(x, y) ≥ γ are identified as aligned. For those x appearing in multiple aligned pairs, we select the pair with the highest score.
To decide the best threshold, we first compute the scores of all candidates and sort them into an ordered sequence. Next, we compute F1 score by setting γ to each middle point of two consecutive scores and find the optimal γ. This procedure is done on training set.
C XNLI Fine-tuning
The fine-tuning hyperparamter setting is shown in Table 9. We searched the learning rate among {1e5, 3e-5, 5e-5, 7e-5}.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
7
7 A2. Did you discuss any potential risks of your work? Our research is fundamental. So, it will not cause much social impact.
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? 4 and 5
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? 4.2
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? 4.2 and appendix C
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 4
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 4
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Exploring DAP’s full capability is left for future work. Besides, RTL task is not the only possible tokenalignment task for our DAP framework. Other objectives based on token representations are also worth investigating. The best objective form is still under research