DAMP: Doubly Aligned Multilingual Parser for Task-Oriented Dialogue
Year: 2023

Abstract:
Modern virtual assistants use internal semantic parsing engines to convert user utterances to actionable commands. However, prior work has demonstrated multilingual models are less robust for semantic parsing compared to other tasks. In global markets such as India and Latin America, robust multilingual semantic parsing is critical as codeswitching between languages is prevalent for bilingual users. In this work we dramatically improve the zero-shot performance of a multilingual and codeswitched semantic parsing system using two stages of multilingual alignment. First, we show that contrastive alignment pretraining improves both English performance and transfer efficiency. We then introduce a constrained optimization approach for hyperparameter-free adversarial alignment during finetuning. Our Doubly Aligned Multilingual Parser (DAMP) improves mBERT transfer performance by 3x, 6x, and 81x on the Spanglish, Hinglish and Multilingual Task Oriented Parsing benchmarks respectively and outperforms XLM-R and mT5-Large using 3.2x fewer parameters.1

heading: 1 Introduction
text: Task-oriented dialogue systems are the backbone of virtual assistants, an increasingly common direct interaction between users and Natural Language Processing (NLP) technology. Semantic parsing converts unstructured text to structured representations grounded in task actions. Due to the conversational nature of the interaction between users and task-oriented dialogue systems, speakers often use casual register with regional variation. Such variation is an essential challenge for the inclusiveness and reach of virtual assistants which aim to serve a global and diverse userbase (Liu et al., 2021).
Work partially done during an internship at Google. 1We release code for our constrained optimization tech-
nique on GitHub and finetuned T5 models on HuggingFace.
In this work, we are motivated by a common form of variation for bilingual speakers (Doğruöz et al., 2021): codeswitching. Codeswitching occurs in two forms which both affect task-oriented dialogue. Inter-sentential codeswitching is when multilingual users make whole requests in different languages within a single dialogue:
Play all rap music on my iTunes Toca toda la música rap en mi iTunes
Intra-sentential codeswitching appears when the user switches languages during a single query:
Play toda la rap music en mi iTunes
Both forms are used by bilingual speakers (Joshi, 1982; Dey and Fung, 2014) and cause location, language preference, and even language identification to be unreliable mechanisms for routing requests to an appropriate monolingual system (Barman et al., 2014). This makes zero-shot codeswitching performance an aspect of system robustness instead of a way to reduce annotation costs.
However, zero-shot structured prediction and parsing is still a challenge for state-of-the-art multilingual models (Ruder et al., 2021), highlighting the need for improved methods beyond scale to achieve this goal. Fortunately, as a fundamental property of the task, these linguistically diverse inputs are grounded in a shared semantic output space. Each of the above outputs corresponds to:
[play_music:[genre:rap][platform:iTunes]]
This grounded and shared output space makes explicit alignment across languages especially attractive as a mechanism for cross-lingual transfer.
We propose using both contrastive alignment pretraining and a novel constrained adversarial finetuning method to perform double alignment, shown in Figure 1. Our Doubly Aligned Multilingual Parser (DAMP) achieves strong zero-shot performance on both multilingual (inter-sentential) and
3586
intra-sentential codeswitched data, making it a robust model for bilingual users without harming English performance. We contribute the following:
1. Alignment Pretraining Effectiveness: We show that multilingual BERT (mBERT) has poor transferability for both categories of codeswitched data. Contrastive alignment, however, pretrained with cross-lingual bitext data dramatically improves English, multilingual, and intra-sentential codeswitched semantic parsing performance.
2. Constrained Adversarial Alignment: We propose utilizing domain adversarial training to further improve alignment and transferability without labeled or aligned data. We introduce a novel constrained optimization method and demonstrate that it improves over prior domain adversarial training algorithms (Sherborne and Lapata, 2022) and regularization baselines (Li et al., 2018; Wu and Dredze, 2019). Finally, we highlight the advantages of pointer-generator networks with explicit alignment by showing that pretrained decoders lead to accidental translation (Xue et al., 2021).
3. Interpreting Alignment Improvements: Additionally, we find the improved parsing ability of DAMP is driven by a 6x improvement in prediction accuracy of the initial intent. Finally, we measure improvements in alignment using a post-hoc linear probe on language prediction in addition to qualitative analysis of embedding visualizations.

heading: 2 Related Work
text: Multilingual Language Model Alignment Massively multilingual transformers (MMTs) (Pires et al., 2019; Conneau et al., 2020a; Liu et al., 2020; Xue et al., 2021) have become the de-facto basis for multilingual NLP and are effective at intrasentential codeswitching as well (Winata et al., 2021). While prior work has studied explicit alignment of individual embeddings (Artetxe et al., 2018; Artetxe and Schwenk, 2019), MMTs appear to implicitly perform alignment within their hidden states (Artetxe et al., 2020; Conneau et al., 2020b).
MMTs are remarkably robust for multilingual and intra-sentential codeswitching benchmarks (Aguilar et al., 2020; Hu et al., 2020; Ruder et al., 2021). However, the gap between performance on the training language and zero-shot targets is larger in task-oriented parsing benchmarks (Li et al., 2021; Agarwal et al., 2022; Einolghozati et al., 2021), similar to the large discrepancy for other syntactically intensive tasks (Hu et al., 2020).
Our work applies the pretraining regime from Hu et al. (2021), which adds multiple explicit alignment objectives to traditional MMT pretraining. We show that this technique is effective both for semantic parsing, a new task, and intra-sentential codeswitching, a new linguistic domain.
Domain Adversarial Training The concept of using an adversary to remove undesired features has been discovered and applied separately in trans-
fer learning (Ganin et al., 2016), privacy preservation (Mirjalili et al., 2020), and algorithmic fairness (Zhang et al., 2018a). When applying this technique to transfer learning, Ganin et al. (2016) term this domain adversarial training.
Due to its effectiveness in domain transfer learning, multiple works have studied applications of domain adversarial learning to cross-lingual transfer (Guzman-Nateras et al., 2022; Lange et al., 2020; Joty et al., 2017). Most relevant, Sherborne and Lapata (2022) combine a multi-class language discriminator with translation loss to improve crosslingual transfer.
In this space, we contribute the 4 following novel findings. Firstly, we show that binary discrimination is more effective than multi-class discrimination and provide intuitive reasoning for why this is true despite the inherently multi-class distribution of multilingual data. Secondly, we show that adversarial alignment can increase the accidental translation phenomena (Xue et al., 2021) in models with pretrained decoders. Thirdly, we show that tokenlevel adversarial discrimination improves transfer to intra-sentential codeswitching. Finally, we remove the challenge of zero-shot hyperparameter search with a novel constrained optimization technique that can be configured a priori based on our alignment goals.
Preventing Multilingual Forgetting Beyond adversarial techniques, prior work has used regularization to maintain multilingual knowledge learned only during pretraining. Li et al. (2018) shows that penalizing distance from a pretrained model is a simple and effective technique to improve transfer. Using a much stronger inductive bias, Wu and Dredze (2019) freezes early layers of multilingual models to preserve multilingual knowledge. This leaves later layers unconstrained for task specific data. We show that DAMP outperforms these baselines, the first comparison of traditional regularization to adversarial cross-lingual transfer.

heading: 3 Methods
text: We utilize two separate stages of alignment to improve zero-shot transfer in DAMP. During pretraining, we use contrastive learning to improve alignment amongst pretrained representations. During finetuning, we add double alignment through domain adversarial training using a binary language discriminator and a constrained optimization approach. We apply these improvements to the en-
coder of a pointer-generator network that copies and generates tags to produce a parse.

heading: 3.1 Baseline Architecture
text: Following Rongali et al. (2020), we use a pointer-generator network to generate semantic parses. We tokenize words [w0, w1 . . . , wm] from the labeling scheme into sub-words [s0,w0 , . . . , sn,w0 , s0,w1 . . . , sn,wm ] and retrieve hidden states [h0,w0 , . . . ,hn,w0 ,h0,w1 . . . ,hn,wm ] from our encoder. We use the hidden state of the first subword for each word to produce word-level hidden states:
[h0,w0 ,h0,w1 . . . ,h0,wm ] (1)
Using 1 as a prefix, we use a randomly initialized auto-regressive decoder to produce representations [d0,d1 . . . ,dt]. At each action-step a, we produce a generation logit vector using a perceptron to predict over the vocabulary of intents and slot types ga and a copy logit vector for the arguments from the original query ca using similarity with Eq. 1:
ga = MLP (da) (2)
ca = [d ⊤ a h0,w1 ,d ⊤ a h0,w1 , . . .d ⊤ a h0,wm ] (3)
Finally, we produce a probability distribution pa across both generation and copying by applying the softmax to the concatenation of our logits and optimize the negative log-likelihood of the correct prediction a′:
pa = σ([ga; ca]) (4)
Ls = −log(paa′) (5) Intuitively, the pointer-generator limits the model to generating control tokens and copying input tokens. This constraint is key for cross-lingual generalization since our decoder is only trained on English data. Even for models which are pretrained for multilingual generation, finetuning on English data alone often leads to accidental translation (Xue et al., 2021), where generation occurs in English regardless of the input language.
The pointer-generator guarantees that our generations will use the target language even for languages it was never trained on. We show that this is essential for DAMP in in Section 5.3, as improved alignment otherwise exacerbates accidental translation by removing the decoders ability to distinguish the input language during generation.

heading: 3.2 Alignment Pretraining
text: We evaluate the contrastive pretraining process AMBER introduced by Hu et al. (2021) for semantic parsing. AMBER combines 3 explicit alignment objectives: translation language modeling, sentence alignment, and word alignment using attention symmetry. These procedures aim to make semantically aligned translation data, known as bitext (Melamed, 1999), similarly aligned in the representation space used by the model.
Translation language modeling was originally proposed by Conneau and Lample (2019). This technique is simply traditional masked language modeling, but uses bitext as input and masking tokens in each language. Since translations of masked words are often unmasked in the bitext, this encourages the model to align word and phrase level representations so that they can be used interchangeably across languages.
Sentence alignment (Conneau et al., 2018) directly optimizes similarity of representations across languages using a siamese network training process. Given an English sentence with pooled representation ei, the model maximizes the negative log-likelihood of the probability assigned to true translation t′ compared to a batch of possible translations B:
L(ei, t ′, N)sa = − log
( e⊤i t
′ ∑
ti∈B e ⊤ i ti
) (6)
Finally, AMBER encourages word level alignment by optimizing with an attention symmetry loss (Cohn et al., 2016). For attention head h ∈ H , a sentence in language S, and its translation in language T , the similarity of the cross-attention matrices AhS→T and A h T→S is maximized:
L(S, T ) = 1− 1 H
∑
h∈H
tr(Ah⊤S→TA h T→S)
min(M,N) (7)
Together, these procedures provide signals which encourage the encoder to represent inputs with the same meaning similarly at several levels of granularity, regardless of which language they occur in.

heading: 3.3 Cross-Lingual Adversarial Alignment
text: However, this alignment across languages can be lost during finetuning. Since procedures such as those used in AMBER rely on manually aligned data, which is rare for downstream tasks, they are
inapplicable for preventing misalignment during finetuning.
Therefore, we instead build on the domain adversarial training process of Ganin et al. (2016) to maintain and improve alignment during finetuning. First, we use a token-level language discriminator as an adversary to maintain word level alignment across languages. We show that multi-class discrimination used in prior work allows for equilibria which are inoptimal for transfer. Instead, we propose treating all languages not found in the training data as a single negative class. Finally, we introduce a general constrained optimization approach for adversarial training and apply it to cross-lingual alignment.
Token-Level Discriminator Similar to Ganin et al. (2016), we train a discriminator to distinguish between in-domain training data and unlabeled outof-domain data. Our method assumes access to labeled training queries in one language, in this case English, and unlabeled queries in multiple other languages which target the same intents and slots. Data is sampled evenly from all languages to create an adversarial dataset with equal amounts of each language.
We use a two-layer perceptron to predict the probability p = P (E|h0,wn) that a token with true label y is English or Non-English given hidden representations from Eq. 1. Our discriminator loss is traditional binary cross-entropy loss:
Ld = −(y log(p) + (1− y) log(1− p)) (8)
Since it is more difficult to discriminate between similar points, domain adversarial training uses the loss of the discriminator as a proxy for alignment. When alignment with the training language improves, so does the cross-lingual transfer to unseen languages.
Prior work using domain adversarial training for multilingual robustness (Lange et al., 2020; Sherborne and Lapata, 2022) performs multi-class classification across all languages and uses the negative log-likelihood of the correct class as the loss function. While using a separate class for each language is natural, it breaks the equivalence between maximizing the discriminator loss and aligning unlabeled and labeled data. With a multi-class discriminator, the generator can instead be rewarded for aligning across unlabeled languages even when this does not benefit transfer from the labeled source.
To illustrate this misaligned reward, suppose we have labeled data in English and unlabeled data in both Spanish and French. The goal of the multiclass adversary is to predict English, Spanish, or French for each token while the encoder is to minimize the ability of the adversary to recover the correct language. Consider the token "dormir", which translates from both Spanish and French to the English "to sleep". In the multi-class setting, the encoder can maximize the adversarial reward by aligning the Spanish "dormir" to the French "dormir", which is simple since they are cognates, without improving alignment with the English "to sleep" at all. In this extreme example, the multiclass loss is likely to lead to a solution which does not improve alignment with the labeled data, in this case English, at all.
Using a binary "English" vs. "Non-English" classifier removes these inoptimal solutions. Since both Spanish and French are now labeled "NonEnglish", the encoder has no direct incentive to align the two unlabeled languages. Instead, the encoder must align both French and Spanish to the labeled English data to the maximize the adversarial reward. Since transferability relies on improved alignment with the labeled data, we expect this loss function to lead to better transfer results.
Constrained Optimization Traditionally, domain adversarial training uses a gradient reversal layer (Ganin et al., 2016) to allow the generator to maximize adversary loss Ld weighted by hyperparameter λ while minimizing task loss Ls. For the generator, this is effectively equivalent to optimiz-
ing a linear combination of the terms:
L = Ls − λLd (9)
Selecting a schedule for λ presents a challenge in the zero-shot setting. Since the reverse validation procedure used to select the λ schedule by Ganin et al. (2016) assumes only one target domain, multilingual works such as Sherborne and Lapata (2022) opt to simply perform a linear search using the in-domain development set s. This approach ignores transfer performance entirely when weighing adversary loss. Instead, we propose a novel constrained optimization method which balances adversarial and task loss automatically using a constraint derived from first-principles.
Our goal is to obtain token representations that are exactly aligned across languages. Any well-fit adversary will predict English with P = 0.5 on such data and receives a loss of 0.3 since it cannot perform better than chance. In equilibrium, the generator cannot increase loss above 0.3 since the adversary can simply predict P = 0.5 for all inputs regardless of the ground truth labels.
This reasoning provides us a clear constraint. In alignment, the Ld should be no less than 0.3, which we call ϵ. We then optimize the task loss Ls while enforcing this constraint. We do so with minimal additional computation cost and using backpropagation alone with the differential method of multipliers (Platt and Barr, 1987). The differential method of multipliers first relaxes the constrained problem to its Lagrangian dual:
L = Ls + λ(ϵ− Ld) (10)
Unlike Sherborne and Lapata (2022), this lets us treat λ as a learnable parameter and optimize it to maximize the value of λ(ϵ − Ld) with stochastic gradient ascent. In plain terms, our optimization increases the value of λ when ϵ > Ld and decreases it when ϵ < Ld. This produces a schedule for λ which weighs the adversarial penalty only when it is accurate. In Figure 3, we show how λ evolves throughout training to maintain the constraint.

heading: 4 Experiments
text: We evaluate the effects of our techniques on three benchmarks for task-oriented semantic parsing with hierarchical parse structures. Two of these datasets evaluate robustness to intra-sentential codeswitching (Einolghozati et al., 2021; Agarwal
et al., 2022) and the third uses multilingual data to evaluate robustness to inter-sentential codeswitching (Li et al., 2021). Examples are divided as originally released into training, evaluation, and test data at a ratio of 70/10/20.

heading: 4.1 Datasets
text: Multilingual Task Oriented Parsing (MTOP) Li et al. (2021) introduced this benchmark to evaluate multilingual transfer for a difficult compositional parse structure. The benchmark contains queries in English, French, Spanish, German, Hindi, and Thai. Zero-shot performance on this benchmark is a proxy for robustness to intersentential codeswitching. Each language has approximately 15,000 total queries which cover 11 domains with 117 intents and 78 slot types.
Hindi-English Task Oriented Parsing (CST5) Agarwal et al. (2022) construct a benchmark of Hindi-English intra-sentential codeswitching data using the same label space as the second version of the English Task Oriented Parsing benchmark (Chen et al., 2020). As part of preprocessing, we use Zhang et al. (2018b) to identify and transliterate Romanized Hindi tokens to Devana-
gari. There are 125,000 in English and 10,896 queries in Hindi-English which cover 8 domains with 75 Intents and 69 Slot Types.
Codeswitching Task Oriented Parsing (CSTOP) Einolghozati et al. (2021) is a benchmark of Spanish-English codeswitching data. While the dataset was released with a corresponding English dataset in the same label space, that data is now unavailable. Therefore, we construct an artificial dataset in the same label space using Google Translate on each segment of the structured Spanish-English training data2. The resulting English dataset is not human validated and therefore noisy. This is a limitation, but is necessary to estimate of zero-shot transfer from English to Spanish-English codeswitching due to the limited release of CSTOP. The resulting dataset has 5,803 queries in both English and Spanish-English which cover 2 domains with 19 Intents and 10 Slot Types.

heading: 4.2 Results
text: We use the same hyperparameter configurations for all settings. The encoder uses the mBERT architecture (Pires et al., 2019). The decoder is a randomly initialized 4-layer, 8-head vanilla transformer for comparison with the 4-layer decoder structure used in Li et al. (2021). We use AdamW and optimize for 1.2 million training steps with early stopping using a learning rate of 2e−5, batch size of 16, and decay the learning rate to 0 throughout the training. We train on a Cloud TPU v3 Pod for approximately 4 hours for each dataset. For all adversarial experiments, we use the unlabeled queries from MTOP as training data for our discriminator and a loss constraint ϵ of 0.3 as justified in 3.3.
The English data from each benchmark is used for training and early stopping evaluation. We report Exact Match (EM) accuracy on all test splits. In all tables, results that significantly (p = 0.05) improve over all others are marked with a † using the bootstrap confidence interval (Dror et al., 2018).
MTOP In Table 1, we report the results of our training procedure with mBERT, AMBER, and DAMP compared to existing baselines from prior work: XLM-R with a pointer-generator network (Li et al., 2021), MT5 (Xue et al., 2021) and byT5 (Xue et al., 2022). For both T5 variants, we train with the hyperparameters described in Nicosia et al. (2021).
2We include the parse brackets during translation to preserve parse structure: Google Translate Documents
Despite finetuned mBERT being a strong baseline for other tasks (Wu and Dredze, 2019; Aguilar et al., 2020; Liang et al., 2020; Hu et al., 2020; Ruder et al., 2021), it is ineffective at cross-lingual transfer for compositional semantic parsing achieving an average multilingual accuracy of 0.5.
The AMBER pretraining process significantly improves over mBERT accuracy for all languages to an average of 23.6. Average accuracy across the 5 Non-English languages improves by 47x. English accuracy also improves to 84.2 from 78.6, instead of suffering negative transfer (Wang et al., 2020).
DAMP further improves average accuracy across languages over AMBER by 1.8x to 42.2, outperforming both similarly sized models (byT5-Base; +34.2, mT5-Base; +16.1) and models three times its size (mT5-Large; +10.8, XLM-R; +3.4). mT5XXL maintains state-of-the-art performance of 55.1 but requires 33x more parameters and multiple GPUs for inference, which increases latency and compute cost.
Adversarial alignment improves performance in each language by at least 10 points, with Hindi and Thai, the most distant testing languages from English, having the largest improvements of +20.7 and +26.5 respectively. DAMP improves over the mBERT baseline by 84x without architecture changes or additional inference cost.
CST5 & CSTOP In Table 2, we report the results on both intra-sentential codeswitching benchmarks. For Hindi-English, we compare the MT5-small and MT5-XXL baselines from Agarwal et al. (2022).
AMBER again leads to a performance improvement over mBERT for both CST5 and CSTOP, across English (+1.4, +5.5) and codeswitched (+12.9, +52.4) data. DAMP also further improves transfer results (+3.8, +1.0) over AMBER at the
cost of small losses in English performance (-0.2, -0.7). DAMP achieves a new state-of-the-art of 20.5 on zero-shot transfer for CST5, outperforming even MT5-XXL (20.3). Since both alignment stages have word-level objectives, we hypothesize that the word-level inductive bias provides benefits for intra-sentential codeswitching despite lacking explicit supervision for it.

heading: 5 Adversarial Baseline Comparison

heading: 5.1 Adversary Ablation
text: In Table 3, we isolate the effects of our contributions to domain adversarial training with an ablation study. While all adversarial variants improve transfer results, we see that using a binary adversary and our constrained optimization technique are both mutually and independently beneficial to adversarial alignment. Notably, DAMP improves over the unconstrained multi-class adversarial technique used in Sherborne and Lapata (2022) by 9.9, 6.4, and 0.9 EM accuracy points on MTOP, CST5, and CSTOP respectively.

heading: 5.2 Regularization Comparison
text: We also compare adversarial training to regularization techniques used in cross-lingual learning. We experiment with freezing the first 8 layers of the encoder (Wu and Dredze, 2019) and using the L1 and L2 norm penalty (Li et al., 2018). Adversarial learning outperforms these baselines on MTOP and CSTOP while model freezing and L2 norm penalization outperform adversarial learning on CST5. However, adversarial learning is the only method that improves across all benchmarks.

heading: 5.3 Pretrained Decoder Comparison
text: Finally, we evaluate whether our constrained adversarial alignment technique offers similar benefits to models with pretrained decoders due to their natural advantage in generation tasks. We find that adversarial training does worse than the plain mT5 model (-9.6). Upon inspection, adversarial alignment causes this drop by exacerbating accidental translation (Xue et al., 2021), where the output for Non-English input is translated to English.
For example, the expected output for “Merci d’envoyer la ligne de travail“ is “[IN:SEND_MESSAGE [SL:GROUP travail]]“. While the unaligned model produces the incorrect parse “[IN:SEND_MESSAGE [SL:RECIPIENT la ligne de travail]]“, the aligned model pro-
duces the correct parse translated to English “[IN:SEND_MESSAGE [SL:GROUP work]]“. In DAMP, the pointer-generator fundamentally prevents accidental translation.
We confirm this in mT5 by reformatting the decoding task in a pointer format, where the correct output in the above example would be “[IN:SEND_MESSAGE [SL:GROUP <pt-5>]]“. This makes accidental translation impossible, and adversarial alignment again improves performance in this variant for MTOP and CSTOP. However, the mT5 decoder struggles to adapt to this task, making overall performance worse than DAMP.

heading: 5.4 Improvement Analysis
text: Since exact match accuracy is a strict metric, we analyze our improvements with qualitative analysis. We examine examples that DAMP predicts correctly but AMBER and mBERT do not. We then randomly sample 20 examples from each language for manual evaluation.
Improvements in intent prediction are a large portion of the gain. If intent prediction fails, the rest of the auto-regressive decoding goes awry as the decoder attempts to generate valid slot types for that intent. We report intent prediction results across the test dataset in Table 4.
In general, these improvements follow a trend from nonsensical errors to reasonable errors to correct. For example, given the French phrase “S’il te plait appelle Adam.” meaning “Please call Adam."", mBERT predicts the intent QUESTION_MUSIC, AMBER predicts GET_INFO_CONTACT, and DAMP predicts the correct CREATE_CALL.
Within the slots themselves, the primary improvements noted in DAMP are more accurate placement articles and prepositions such as "du", "a", "el", and "la" inside the slot boundaries, which is of arguable real world importance.
We present the full sample of examples used for this analysis in Tables 5-9 in the Appendix.

heading: 6 Alignment Analysis
text: We analyze how well our alignment goals are met using two methods in Figure 1. First, we use a twodimensional projection of the resulting encoder embeddings to provide a visual intuition for alignment. Then, we provide a more reliable quantitatively evaluate alignment using a post-hoc linear probe.

heading: 6.1 Embedding Space Visualization
text: In Figure 1, we visualize the embedding spaces of each model variant on each MTOP test set using Universal Manifold Approximation and Projection (UMAP) (McInnes et al., 2018). Our visualization of mBERT provides a strong intuition for its poor results, as English and Non-English data form linearly separate clusters even within this reduced embedding space. By using AMBER instead, this global clustering behavior is removed and replaced by small local clusters of English and Non-English data. Finally, DAMP produces an embedding space with no clear visual clusters of Non-English data without English data intermingled.

heading: 6.2 Post-Hoc Probing
text: We evaluate improvements to alignment quantitatively. While Sherborne and Lapata (2022) reports the performance of the training adversary as evidence of successful training, this method has been shown insufficient due to mode collapse during training (Elazar and Goldberg, 2018; Ravfogel et al., 2022). Therefore, we train a linear probe on a frozen model after training for each variant using 10-fold cross-validation.
Supporting the visual intuition, probe performance decreases with each stage of alignment. On mBERT, the discriminator achieves 98.07 percent accuracy indicating poor alignment. AMBER helps, but the discriminator still achieves 93.15 percent accuracy indicating the need for further removal. DAMP results in a 23.62 point drop in discriminator accuracy to 69.53. This is still far above chance despite our training adversary converging to close-to-random accuracy. This indicates both the need for post-hoc probing and the possibility of further alignment improvements.

heading: 7 Conclusions
text: In this work, we introduce a Doubly Aligned Multilingual Parser (DAMP), a semantic parsing training regime that uses contrastive alignment pretraining and adversarial alignment during fine-tuning
with a novel constrained optimization approach. We demonstrate that both of these stages of alignment benefit transfer learning in semantic parsing to both inter-sentential (multilingual) and intrasentential codemixed data, outperforming both similarly sized and larger models. We analyze the effects of DAMP, comparing our proposed alignment method broadly to prior both adversarial techniques and regularization baselines, and its generalizability, with applications to pretrained decoders. Finally, we interpret the impacts of both stages of alignment through qualitative improvement analysis and quantitative probing.
Importantly, DAMP shows that alignment in both pretraining and finetuning can outperform larger models pretrained on more data. This offers an orthogonal improvement to the current scaling paradigm, supporting the idea that current multilingual models underutilize available bitext (Reid and Artetxe, 2022). In cases where bitext is unavailable, our work shows that alignment still possible via adversarial procedures. By releasing our simplified constrained optimization approach for multilingual adversarial alignment, we aim to simplify and improve the application of such approaches for future work.

heading: 8 Limitations
text: This work only carries out experiments using English as the base training language for domain adversarial transfer. It is possible that domain adversarial transfer has a variable effect depending on the training language from which labeled data is used. Additionally, while typologically and regionally diverse, all but one language used in our evaluation is of Indo-European origin.

heading: 9 Acknowledgements
text: We are thankful to Hongxin Zhang, Caleb Ziems, and the anonymous reviewers from Google, ACL Rolling Review, and the ACL Main Conference for their helpful feedback.

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section 8

heading: 3 A2. Did you discuss any potential risks of your work?
text: We discuss in our limitations section (8) the possibility that our work does not work across broader multi-lingual gaps and could exacerbate the cross-lingual divide. Other than this limitation, our work performs a previously established tasks so poses no major additional risk.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract and Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 4.1 covers datasets used.

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 4.1 cites these datasets.
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? All datasets are released under the Creative Commons license which is permissive of our research use.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 4.1 - we use all datasets according to their original intended use case of training and evaluating task-oriented dialogue systems.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. We did not produce any new datasets.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. We did not produce any new datasets.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 4.1 provides descriptive statistics of each dataset used.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C 3 Did you run computational experiments? Section 4.2

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 4.2

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 4.2

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? We state that we report single run results with pairwise bootstrap tests. Details are in the caption of each table of statistics.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: By releasing our simplified constrained optimization approach for multilingual adversarial alignment, we aim to simplify and improve the application of such approaches for future work