Transformer Architecture Search for Improving Out-of-Domain Generalization in Machine Translation
Authors: Yiheng He
Ruiyi Zhang
Sai Ashish Somayajula
Pengtao Xie
Venue: Accepted by TMLR ()

Abstract: Interest in automatically searching for Transformer neural architectures for machine translation (MT) has been increasing. Current methods show promising results in in-domain settings, where training and test data share the same distribution. However, in real-world MT applications, it is common that  the test data has a different distribution than the training data. In these out-of-domain (OOD) situations, Transformer architectures optimized for the linguistic characteristics of the training sentences struggle to produce accurate translations for OOD sentences during testing. To tackle this issue, we propose a multi-level optimization based method  to  automatically  search for  neural architectures that possess robust OOD  generalization capabilities. During the architecture search process, our method automatically synthesizes approximated OOD MT data, which is used to evaluate and improve the architectures' ability of generalizing  to OOD scenarios.  The generation of approximated OOD data and the search for optimal architectures are executed in an integrated, end-to-end manner. Evaluated across multiple datasets, our method demonstrates strong OOD generalization performance, surpassing state-of-the-art approaches.

Published in Transactions on Machine Learning Research (12/2024)
Transformer Architecture Search for Improving Out-of- Domain Generalization in Machine Translation
Yiheng He
yihenghe@@berkeley.ddi edu
UC San Diego
Ruiyi Zhang UC San Diego
ruz0488 edu
Sai Ashish Somayajula
ssomayaj@ucsdd.o edu
UC San Diego
Pengtao Xie
p Gucsd. edu
UC San Diego
Reviewed on OpenReview: https: // openreview. net/ forum? id= 8M6cn61 frJ
Abstract
Interest in automatically searching for Transformer neural architectures for machine trans- lation (MT) has been increasing. Current methods show promising results in in-domain set- tings, where training and test data share the same distribution. However, in real-world MT applications, common that the test data has a different distribution than the training data. In these out-of-domain (OOD) situations, Transformer architoctures optimized for the linguistic characteristics of the training sentences struggle produce accurate translations for OOD sentences during testing. To tackle this issue, propose multi-level optimiza- tion based method to automatically search for neural architectures that possess robust OOD generalization capabilities. During the architecture search process, our method automati- cally synthesizes approximated OOD MT data, which is used to evaluate and improve the architectures' ability of generalizing to OOD scenarios. The generation approximated OOD data and the search for optimal architectures are execuled in an integrated, end-to- end manner. Evaluated across multiple datasets, our method demonstrates strong OOD generalization performance, surpassing statc-of-the-art approaches. Our code is publicly available at htttps://gittubb.com wow.://.....ommceegeneeeeeeeeeee wewpangghgeeraasoomeee.t..... nas
Introduction
Machine Translation (MT) is a pivotal area within natural language processing (NLP), enabling the an- tomatic translation texts from one language to another. The emergence of Neural Machine Translation (NMT) (Bahdanau al. 2014) has revolutionized the field by eliminating the need for labor-intensive feature engincering through end-to-cnd training mechanism. Many deep neural networks, such Recurrent Neu- ral Networks (RNNs) Bahdanau et al. (2014) and Long Short-Term Memory (LSTM) Hochreiter & Schmid- huber 1997 networks, have propelled advancements in NMT. Notably, the Transformer model Vaswani et al. 2017), which leverages self-attention to capture long-range dependency between tokens, has signifi- cantly enhanced MT performance. Despite these advancements, current NMT models are manually designed,
"Equal contribution.

---

Published in Transactions on Machine Learning Research (12/2024)
process that demands significant time, resources, and expert knowledge, involving extensive trial-and-error for optimization. There may be superior design alternatives that go unexplored, hindered by constraints on time and resources, as well as potential human biases.
To tackle this issue, Neural Architecture Search (NAS) Liu et al. 2018;: : Chen et al. 2019 methods have been developed to autonomously identify the most effective Transformer architectures for machine transla- tion. For example, Evolved Transformer [oo et al. (2019), which is an evolutionary algorithm based NAS method, achieves state-of-the-arr MT performance. To mitigate the high computational costs incurred by the evolutionary algorithm in Evolved Transformer, Zhao et al. (2021) proposed differentiable neural ar- chitecture search method which has the same space as Evolved Transformer. This approach significantly improves computation efficiency while achieving translation performance comparable to, if not better than, that of Evolved Transformer.
Despite their potential, existing NAS methods encounter a notable obstacle: limited out-of-domain general- ization. Architectures identified through current NAS techniques perform well within the specifie domains and datasets they are optimized for. However, their performance significantly decreases when dealing with texts from domains or styles that differ from those in the training set. The issue stems from the fact that NAS tends to overfit to the peculiarities of the training data, capturing idiosyneratic features that do not necessarily translate to broader linguistic or contextual generalizability. Consequently, when faced with novel or diverse linguistic data outside the narrow confines of their training environment, these specialized architectures often struggle to maintain the same level of accuracy and fluency, highlighting a critical gap between comainn-specifi optimization and universal applicability in machine translation tasks.
To address this problem, propose novel NAS framework to scarch for optimal Transformer archi- tecture for MT lasks that achieves superior OOD generalization performance. Our framework is based on multi-level optimization (MLO) Choe et al. (2022) which performs three learning stages end-to-end. In the first stage, we train the weight parameters of a Transformer model, keeping its learnable architecture tentatively fixed, by minimizing the loss on an MT training dataset. In the second stage, we generate an additional dataset that maximizes the distribution discrepancy with the real training data while preserving semantic consistency between paired source and target sentences. This step aims create close approx- imation to OOD data, addressing the limited availability of true OOD data during training. To achieve this, we introduce small, learnable perturbations to the embeddings of source sentences and optimize these perturbations. These adjusted source sentences, paired with their corresponding target sentences from the training dataset, form an auxiliary dataset that serves as synthetic OOD data. In the final stage, we assess the loss of the Transformer model (trained in the first stage) on the OOD data (generated in the second stage) and refine its architecture by minimizing this loss. The three stages are conducted jointly by solving the MLO problem. We conducted comprehensive experiments across both OOD and in-domain MT tasks. Our method significantly outperforms vanilla Transformer and prior NAS-based Transformer architectures. Extensive ablation studies further validate the importance of each component in framework. Moreover, our method only incurs modest increase in computational costs (less than 10%), compared to bascline NAS methods.
To address this challenge, we propose a novel Neural Architecture Search (NAS) framework to identify optimal Transformer architectures for machine translation (MT) tasks with superior out-of-domain (OOD)
generalization performance. Our approach leverages multi-level optimization (MLO) Choe al. (0022) which operates in three integrated learning stages in end-to-end manner.
The major contributions of this paper can be summarized as follows.
We propose a three-level optimization based method that automatically searches for a Transformer architecture to improve the OOD generalization performance in MT.
Our method significantly outperforms both the vanilla Transformer architecture and NAS base- lines on OOD tasks. For instance, our approach enhanced the baseline method's performance by 29% in BLEU score when assessed using the Gnome (English-Igbo) dataset, and achieved a 38% improvement in BLEU score on the Ubuntu (English-Igbo) dataset.
2

---

Published in Transactions on Machine Learning Research (12/2024)
Stage I: Train Transformer weights
Architecture
Evaluate OOD performance
Update Architecture
Real MT data
Stage III
Trained
Stage II: Learn
weights
perturbations
OOD MT data
OOD performance
Figure 1: Overview of our method. It consists of three learning stages which are performed end-to-cnd. In the first stage, we train the Transformer's weight parameters with its architecture tentatively fixed. In the second stage, we generate a synthetic OOD MT dataset by adding learnable perturbations to real MT data. In the third stage, we evaluate the trained Transformer's performance on the OOD data and update architecture by maximizing this performance.
2
Related Works
2.1 Neural Architecture Search (NAS)
The objective of NAS is to autonomously discover neural network architectures that have the potential to outperform those created by humans. In recent years, NAS has witnessed significant. advancements. NAS methods are categorized into three primary types: reinforcement learning (RL)-based, evolutionary algorithms (ase,, and gradient- based. A detailed discussion of RL-based and EA-based NAS methods is provided in Appendix A To address the issue of high computational cost of RL-based and EA-based NAS approaches, researchers have proposed differentiable search techniques Cai et al. (2019); Liu et al. (2018): Xie et al. (0119). These methods conceptualize each potential architecture building blocks, with each block assigned a specific importance weight. The process of architecture search is then simplified to optimizing these weights, task achievable through differentiable optimization techniques like gradient descent. Specifically, many gradient-based NAS methods use one-shot architecture search, in which architecture weights are learned by minimizing loss on a validation dataset while model weights are learned simultancously by minimizing loss on training dataset. This approach significantly enhances computational efficiency compared to RT.-based methods. Recently, differentiable NAS techniques have been used to search for optimal Transformer architectures as well Zhao et al.) (2021). Our method also belongs to the category of gradient-based NAS, but unlike previous approaches, il specifically focuses on searching for architectures that optimize snt-of-domail generalization.
configuration of numerous
2.2 Out-of- oofaain Generalizable Learning
Existing methods for improving the OOD generalizability of ML models can be roughly categorized into two paradigms. The first paradigm learns domain-invariant latent representations so that the discrepancy of different domains is mitigated in the latent space. Domain Invariant Component Analysis Muandet et al. (2013) is a kernel-based algorithm that learns a transformation to minimize the difference between marginal distributions of different domains. Multi-task Autoencoder (MTAE) Ghilary al. (2015) jointly trains an encoder and several domain specific decoders to learn invariant representations with multi-task learning. MuluTCA Grubinger al. (2017) learns domain-invarianl representations by minimizing the discrepancy between domains while maximizing the variance within each domain. EISNet WWan et al. (2020a) learns to generalize across domains by solving momentum metric learning task and self-supervisct auxiliary task.
The second paradigm of methods focuses on generating additional data and leveraging it to improve OOD generalizability. Domain Randomization Tobin et al. (2017) creates synthetic OOD images by randomly combining elements from different environment simulators. GUD Volpi et al. (2018) employs adversarial domain augmentation to iteratively generate fictitious worst-case target distributions. Jigen Carlueci et al.
2019h augments data by transforming it into jigsaw puzzles to simulate OOD scenarios. CrossGrad Shankar

---

Published in Transactions on Machine Learning Research (12/2024)
et al. (2018) uses domain-guided perturbations to synthesize diverse training data. MADA Qiao et al (0020) applies meta-learning to optimize meta parameters by minimizing the loss on augmented data from multiple domains and uses a Wasserstein autoencoder to relax semantic constraints commonly associated with domain augmentation. UMGUD Qiao & Peng [021]] introduces perturbations to imput features and augments labels by injecting randomness derived from these perturbations. Unlike these methods, our approach employs multi-level optimization to integrate additional data simulation with OOD generalizable model training an end-to-end manner. This holistic framework ensures that the simulated data is not only diverse but also highly effective in enhancing the model's robustness to distributional shifts.
2.3 Noisy Embeddings
Tt is worth noting that noisy embedding based data augmentation methods not only benefit OOD gener- alization tasks, mentioned in the previous scction, but enhance model's in-domain capability as well. For instance, FreeLB Zhuu et al. (2020) adds adversarial perturbalions to word embeddingg,, enhancing the performance of Transformer- based models on natural language understanding and commonsense reasoning tasks. WaffleCIIP Roth ct al. (2023) adds random words to image captions during the fine-tuning of vision- language models, yielding performance gains. NAYER Tran et al. (2023) introduces noisy layers to generate perturbed embeddings for image text captions, improving performance on data-free knowledge distillation tasks. BFTSS Somayajula et al. (2023) learns semantically consistent perturbations for embedding vectors, effectively enhancing performance on NLU tasks Wang (2018). NEFTTue Jain et al. (2024) introduces ran- dom noise to the embedding vectors of the training data during the forward pass of fine-tuning, improving the outcome of instruction fine-tuning, Our method employs a novel strategy to generate noisy embed- dings under a multi-level optimization framework, which enhances performance in both out-of-domain and in-domain scenarios.
2.4 Bi-level and Multi-level Optimization
A wide range of machine learning applications leverage bi-level optimization (BLO), which represents the simplest form of multi-level optimization, characterized by a two-tier hierarchy. Specifically, BLO consists of two nested optimization problems with a mutual dependency. These applications include, but are not. limited to, neural architecture search Liu et al. (2018): Zhang et al. (2021), hyperparameter optimization Baydin
ct al. 2017); Feurer et al.] (2015): : Francoschi
al. (2017] 2018); Lorraine et al. (2020); Maclaurin et al.
(2015) reinforcement learning Hong et al. (2020); Konda & Tsitsiklis (1999); Rajeswaran et al. (2020)
data valuation Ren et al. (2020) Shu et al. (0019 : Wang et al. (2020b). meta learning Finn et al. 2001
Rajeswaran et al. (2019). and label correction Zheng et al. 20119.
As BLO gains popularity, the focus has also broadened to multi-level optimization (MLO) involving more complex hierarchical structures Garg et al. (2022); He et al. (2021); Raghu et al. (2021): y Somayajula et al.. (2022]: Such et al. (2020 ; Xie & Du (2022). MLO consists of more than two nested optimization problems with more complicated dependencies. Recent research in this area has shown growing interest in con- structing and optimizing multi-stage machine learning pipelines end-to-end using MLO. However, solving MLO problems poses computational challenges due to its complex structure. [aato et al. (2021) proposed a gradient-based solver. [Choe et al. (2022) developed a software which enables users to compute hyper- gradients within MLO problems with multiple approximation methods easily and efficiently. To cope with high memory and computation costs associated with large-scale multi-level optimization, Choe et al. (2023) developed a distributed framework. In this paper, we propose framework with three nested optimization problems, and solve this MLO problem with gradient-based algorithm.
3 Method
Let W and A denote the weight parameters and learnable architocture of a Transformer model used for machine translation (MT). This model takes a souree sentence as input and generates target sentence. Let Drr_ ((r, 31) 2 denote an MT training dataset where I; is an input source sentence and yi is
4

---

Published in Transactions on Machine Learning Research (12/2024)
the corresponding target sentence. Our framework consists of three learning stages which are performed end-to-end.
3.1 Stage I
In the first stage, we train the weight parameters W of the Transformer with its architecture A tentatively fixed. Given a source sentence 2; from D it is fed into the Transformer which generates a target sentence [(((;;; W, A), where E' is an embedding module for sentences. A teacher-forcing based negative log likeli- hood (NLL) loss used to measure the discrepancy between f(E(x;)) W, and the ground-truth target sentence ye- We learn W by minimizing I defined on each training example. This stage amounts to solving the following optimization problem:
(A) = argminw Σ
(L) 1=1
(1)
where W (A) denotes that the optimal solution W depends on A since W depends on the loss function, which depends on A. Note that A cannot be learned at this stage. Otherwise, the result will be an overfitted solution where A perfectly fits the training data but performs poorly on the test data.
3.2
Stage II
In the second stage, we generate additional MT dataset to learn architecture with robust OOD generalization capabilities. Since real OOD data is unavailable, we create a close approximation of OOD data for use in architecture search in the next stage (see below). For each real training example (I,, 34) E D we first map an input sentence T2 to a sentence embedding with the embodding module E. We add a small learnable perturbation & to E(x;) in way that the perturbed embedding E(x;) + & satisfies the following two conditions. First, the translation y for E(x;) can still be used as a translation for E(x;)+ Second, the perturbed source sentence embeddings S( = (E(x;)+ + should have a large domain discrepancy with the original source sentence embeddings S = (E(a,)) To satisfy the first condition, we use the trained MT model with weights (A) to generate translation f(E(x;) + d;; W (A), and learn to minimize the discrepancy (measured using the NLL I) between f(E(x;) + dc, W (A), A) and yi- To satisfy the second condition, the domain difference between ((6,)) and S, and learn { to maximize M (S ( 1), 8). Let k(-,-) be a kernel function. Given two distributions p and y, their MMD is defined as:
the Maximum Mean Discrepancy (MMD) Gretton et al. (0122) M to measure
[kb + Ey [k. 3/)l 2E
(2)
Given set of sample { drawn from p and set of sample {ur drawn from q, the empirical MMD is calculated as:
m (m 1) En (1), + n(n-1) Em Eny; key U5)
1
mn D Σ} k(
2
We choose to use MMD for mcasuring domain discrepancy bocause it is non-paramctric, requiring no as- sumptions about the underlying distributions, and leverages the kernel trick to efficiently capture complex, high-dimensional data structures.
This stage amounts to solving the following optimization problem:
(A)) = argmin (1) Σ ((f(E(z)) - 5, W (A), XM( -
(4)
where À is a tradeoff parameter.
3.3 Stage III
In the third stage, we evaluate the MT model trained in the first stage on the generated dataset in stage IT ((F(x;) (A), and update the architecture A by minimizing the evaluation loss. This stage
5

---

Published in Transactions on Machine Learning Research (12/2024)
amounts to solving the following optimization problem:
N
minA ((A)
(5)
3.4 A Multi-level Optimization Framework
Putting these pieces logether, we have the following three-level oplimization problem:
minA ((A)
+
s.t. {A)( = argmin 16.12
Σ ([((((()))) + +;; (A), 3)) (S(((())))
(6)
W (A) = argminw E Σ (.
The three stages are mutually dependent on each other. The out (*A)) of the first stage is the input of the loss function in the second stage. The outputs of the first two stages, including W (*(A) and {6t (A)) are the inputs to the loss function in the third stage. The optimization variable A in the third stage used to define the loss functions in the first and second stage. By solving the three interdependent optimization problems together, we can perform the three learning stages end-to-end.
3.5 Search Space and Search Method
We set the the candidate operation set our experiments following [oo al. (2019) Zhao et al. (2021) which detailed in Appendix B Finally, we apply dropout (Srivastava al. (2014) to the output of an operation, then add this result to the original input to form residual connection He et al. (0016). After
this, layer normalization Ba et al. (2016) is applied to the combined output.
We adopt a differentiable search method Liu et al. (2018), where each candidate operation is associated with a selection variable representing how likely this operation is going to be selected into the final architecture. The architecture A represented by the collection of selection variables and architecture search amounts lcarning these variables. After learning, operations associated with the highest-valued selection variables are preserved to construct the final architecture.
3.6 Optimization Algorithm
We develop a hypergradient based method to solve the problem in Eq. 10). First, we approximate W*(A) using one-step gradient descent update of W w.r.t the loss function in the first stage:
(A) R5 W' = W The Vw EN ((f(E(x)); W).)..)
(7)
where The a learning rate. Then we plug the approximation (*AA) W into the loss function in the second stage, and approximate (A) using one-step gradient descent update of &, w.rtt the approximated loss function:
(A) =z 6 71 V., (IU(((),) ( + 8,; (", A)) - (M(((()))))))
(8)
where V/ is a learning rate. Finally, we plug the approximation (A) Rs && into the loss function in the third stage and update by gradient descent:
N A 4 A- Na V Σ
(F((((.)) W A), y.)
(9)
where is a learning rale. After A is updated, it is plugged into Eq. 7 and Eq. S). yielding a new W and a new 8. These update steps iterate until convergence, yielding the final learned A' xe A", as summarized in Algorithm [ We provide additional details about our optimization algorithm Appendix [ including the detailed computation of the gradient in Eq. 9 and a discussion on the convergence properties of the method.

---

Published in Transactions on Machine Learning Research (12/2024)
Algorithm Optimization Algorithı
Initialize a model with weights W and architecture A. while not converged do
1. Update weights W with equation
2. Update perturbation 8 with equation
3. Update architecture A with equation
end while
Derive the final architecture based on learned A.
4 Experiments
4.1 Datasets
We evaluate OOD generalization performance on low-resource languages, including English-Igho (En-Ig). English-Hausa (En-Ha), and English-Irish (En-Ga) language pairs. Igbo, part of the Niger-Congo language family, incorporates diacrities, presenting challenges for MT tasks Orife (2018) Dossou & Emeeze (2021)
Hausa, a Chadic language within the Afroasiatic phylum, presents unique linguistic features. Irish, Goidelic language of the Celtic family, employs an array of grammatical mutations and a VSO (verb-subject-ohjeet
word order, which pose challenges for machine translation tasks Dhonnchadha et al. (2003). J. Following Ahia et al. (2021), we obtain training data for En-Ig, En-Ha and En-Ga from the CCMatrix parallel corpus Schwenk al. (2019), which offers the largest collection of high-quality, web-based bitexts for machine translation. The test data for En-Ig and En-Ha is from the Gnome and Ubuntu datasets, and the test data for En-Ga. is from the Flores dataset, all considered OOD for CCMatrix. We include detailed description of these datasets in Appendix D
Furthermore, we conduct experiments on high-resource languages, following So et al. 2019 and Zhao et al. 22011, using the following training datasets: 1) WMT18 English-German (Fn-De) without ParaCrawl, consisting of 4.5 million sentence pairs; 2) WMT14 English-French (En-Fr), comprising 36 million sentence pairs; and 3) WMT18 English-Czee Czech (En-Cs) without ParaCrawl, with 15.8 million sentence pairs. The test data for these language pairs is from the WMT- and WMT- WMT-Biomedical datasets, both considered OOD for WMT14 and WMT18. Detailed descriptions can be found in Appendix D
4.2 Baselines
Our method is evaluated against the vanilla Transformer architecture Vaswani et al. (017 We also compare method with differentiable architecture search baselines including DARTS Liu et al. (2018) and PDARTS Chen et al. (2019). that balance performance and computational efficiency. We did not compare with evolutionary algorithm based methods So et al. (2019) since they are computationally very expensive. We include detailed description of baseline methods in Appendix
Our method represents general framework that can be integrated with various differentiable NAS mothods. For example, the search space and search strategy in our method be set to those in either DARTS PDARTS. We use Ours-darts and Ours-pdarts to denote that method is integrated with DARTS and PDARTS respectively.
4.3 Experimental Settings
Search configuration. We follow the settings in Zhao et al. 2021 for architecture search. Both the vanilla DARTS and Ours-darts utilize two identical encoder and two identical decoder layers during the search phase. Each encoder layer consists of 'Self Atlention' + 'Search Node' + 'Search Node', whereas each decoder layer consists of 'Self Attention + *Cross Attention' + 'Search Node + 'Search Node: Only the 'Search Node is scarched, while the architectures of 'Self Attention' and 'Cross Attention' are fixed. Zhao et al.] (2021) empirically shows that architecture search with this configuration yields better results. The layers with searched architecture stacked to construct the final model with six encoder layers and
7

---

Published in Transactions on Machine Learning Research (12/2024)
Methods
En-Ig
En-Ig
En-Ha
En-Ha (Ubuntu)
En-Ga (Flores)
En-De (Chat)
En-De
En-Fr
En-Cs
(Gnome)
(Ubuntu)
(Gnome)
(Biomed)
(Chat)
(Biomed)
Transformer
2.04(0.07) (.03
(.. (.1
(0.20)
(8.74
26.90 (4.08
19.84(0.26)
DARTS
(.26(
0.75
(..00)
26. (0.08)
29.03 55.39 29.03().25)
(..30
(2.62
Ours-darts
(o.0)
(0.07) (.07)
0.83(0.08)
33.19 (02 (0.2)
(0.46)
(0.20)
25.08(0.88)
PDARTS
(.28
(o.1)
0.96
0.49 (0.14)
29.56(u.25)
29. (9.70(1.27)
58.24(u.64)
27. 27.41 (0.22) (3.53
Ours-pdarts (0.7 (o.2) (..8) 1.01 33.33 (0.24) 31.29 (9.93
28. 28.09
24.91 (0.2)(0.28)
Table 1: BLEU scores (mean and standard deviation across three runs) in nine out-of-domain (OOD) generalization experiments, where each column corresponds to an experimental setting: (1-2) The training dataset was En-Tg (CCMatrix) and the test datasets included En-Ig (Gnome) and En-Ig (Ubuntu); (3-4)
The training datasct was En-Ha (CCMatrix) and the test datasets included En-Ha (Gnome) and En-Ha (Ubuntu); (5) The training datasel was En-Ga (CCMatrix) and the lest dataset was En-Ga (Flores); (6-7) The training dataset was En-De (CCMatrix) and the test datasets included En-De (WMT-Chat) and En- De (WMT- : (8) The training dataset was En-Fr (CCMatrix) and the test datasct was En-Fr (WMT-Chat); and (9) The training dataset was En-Os (CCMatrix) and the test dataset was En-Cs (WMT- Biomedical). Double-sided t-tests were conducted between our methods and baselines, with p-values less than 0.05, indicating statistically significant improvements achieved by our methods over baselines.
six decoder layers. PDARTS and Ours-pdarts adopt a progressive learning approach, where the number of encoder and decoder layers increases from 2 to 4 6 through the search process. Simultaneously, the size of the operation set in the encoder layer is reduced from 15 10 to 5, and in the decoder layer from 16 to 11 to 6.
Hyperparameter settings. Following Vaswani et al. (2017), we utilize 6 encoder and decoder layers, a hidden size of 512, a filter size of 2048, and 8 attention heads for vanilla Transformers, DARTS, PDARTS, Ours-darts and Ours-pdarts. For Ours-darts and Ours-pdarts, the radial basis function (RBF) kernel is used to compute maximum mean discrepancy (MMD) used in Stage 11. The tradeoff parameter À is set to 1.5.
For the optimization of W in our methods and baselines, the same hyperparameter settings as Vaswani et al. 2017) are used, including the learning rate and its scheduler, with warm-up steps set to 4000. The Adam Kingma & Ba (2014) optimizer with 31 = 0.9, 83 0.98, and c= 10 used. We increase the learning rate linearly for the first warmup training steps, and decrease it Chereafter proportionally to the inverse square root of the step number. For the optimization of architecture weights A, both our methods and the basclines use the same hyperparamcters following Liu et al. (2018), employing a constant lcarning rate of 3 x 10-4 and a weight decay of 10-3, with the Adam optimizer (31 = 0.9, B2 = 0.98, and e = 10-9) being used. The perturbation o, in our framework, is optimized using an Adam optimizer with 31 = 0.9. 32 = 0.98, and = 10-9 and a constant learning rate of 10-3, 3, We set the dropout rate to 0.1 and employ label smoothing with a value of 0.1 during architocture scarch. We set the batch size to 4096 for all experiments.
The maximum sentence length is set to 256 for all experiments. Tokenization is performed using Moses which is rule-based tokenizer. BLEU Papineni et al. (2002) is used as the evaluation metric. We employ beam search during inference with a beam size of 4 and a length penalty a = 0.6. All the experiments were conducted on Nvidia 110 GPU.
4.4 Results on Out-of-Domail Generalization
We evaluate the OOD generalization performance of our method in nine experiments. In experiment 1 and 2, the training dataset was En-Ig (CCMatrix) and the test datasets included En-Ig (Gnome) and En-Ig (Ubuntu). In experiment 3 and 4, the training dataset was En-Ha (CCMatrix) and the test datasets included En-Ha (Gnome) and En-Ha (Ubuntu). In experiment 5, the training datasct was En-Ga (CCMatrix) and
8

---

Published in Transactions on Machine Learning Research (12/2024)
Methods
En-Fr (WMT)
En-Cs (WMT)
Transformer
38.14
24.22
DARTS
38.42
25.00
Ours-darts
38.85
25.24
PDARTS
39.39
25.33
Ours-pdarts
39.73
25.89
Table 2:
BLEU scores in the third OOD generalization experiment, where the dataset for architecture search was En-De (WMT) and the datasets for retraining and testing included En-Fr (WMT) and En-Cs (WMT).
Methods
En-Ig (CCmatrix)
En-Ha (CCmatrix)
En-De (WMT)
Transformer
52.04
0.59
44.73 1.05
27.27
DARTS
41.95 + 1.04
37.22 + 2.33
27.55
Ours-darts
55.50 + 0.18
47.28 + 0.94
27.84
PDARTS
45.38
1.01
40.34 0.62
28.11
Ours-pdarts
59.23 + 2.09
57.41 + 1.90
28.24
Table 3: BLEU scores of in-domain generalization experiments, where training and test data are from the same dataset. Each of the three datasots, including En-Ig (CCmatrix). En-Ha (CCmatrix), and En-De (WMT), divided into training, validation, and test splits. Models are trained the training split, evaluated the test split (results shown in this table), and the validation split is used for tuning hyperparameters.
the test dataset was En-Ga (Flores). In experiments and 7, the training dataset En-De (CCMatrix) and the test datasets included Fn-De (WMT-Chat) and En-De (WMT-Biomedd)) training dataset was En-Fr (CCMatrix) and the test dataset was En-Fr (WMT-Chat). In experiment 9, the training dataset was En-Cs (CCMatrix) and the Lest dataset was En-Cs (WMT-Bionediial)) Additionally, we evaluate the OOD generalization performance of our method across language pairs. Specifically, the dataset for architecture scarch was En-De (WMT) and the datascts for retraining and testing included En-Fr (WMT) and En-Cs (WMT).
In experiment 8, the
Table and 2 show the results. As can be seen, Ours-darts and Ours-pdarss demonstrate superior per- formance compared to DARTS and PDARTS, respectively. In particular, Ours-pdarts achieves the best performance on 8 out of 9 datasets across all methods. This is because our method automatically generates approximated OOD data (in stage IT) and optimizes the architecture by minimizing the MT loss on this generated data (in stage III). By doing so, our method aims to enhance the architecture's ability to generalize real OOD data. Such mochanism lacking DARTS and PDARTS. Furthermore, Ours-darts and Ours-pdarts surpass the performance of the vanilla Transformer architecture, which further demonstrates the superior OOD generalization capability of methods, due to its mechanism of generating additional data to improve the OOD generalization performance of architectures. In contrast, the manual designed Transformer architecture does not take OOD generalization into account.
4.5 Results on In-Domain Generalization
In addition to the OOD generalization performance, we also evaluated the in-domain generalization perfor- mance of searched architectures, where the training and test data are disjoint splits of the same dataset. Datasets used for this evaluation include WMT18 Gnelish-German (En-De), WMT14 English-Frende (En- Fr), WMT18 English-Czech (En-Cs), CCMatrix English-Igbo (En-Ig) and CCMatrix English-Hausa (En- Ha). Table 3 shows the results. As can be seen, Ours-darts outperforms DARTS; Ours-pdarts outperforms PDARTS; and both of methods outperform vanilla Transformer. This demonstrates the superior per-
9

---

Published in Transactions on Machine Learning Research (12/2024)
formance of methods in scenarios where the domain of language pairs in test data are the same those in the training data. This superiority highlights the robustness of our method to both in-domain and out-of-domain machine translation scenarios.
4.6 Ablation Studies
To better evaluate the effectiveness of individual components in our framework, we perform several ablation studics.
Sensitivity to the tradeoff parameter X. We investigate how the trade-off parameter À in our framework affects downstream performance. À is varied within the set {0.5, 1.0, 1.5, 2.0}. The WMT18 English-German German dataset was used the training data and WMT14 English-French was used as test data. The study conducted using Ours-darts. Figure 2 shows how the BLEU score on the WMT14 English-French test set varies as À increases. As can be seen, a À value in the middle ground yields the best performance. A very small such as 0.5 reduces the contribution from the MMD, making the domain difference between generated data and real training data small. a result, the generated data cannot effectively improve the OOD gencralization performance of the architocture. Conversely, a higher increases the contribution from the MMD, encouraging a larger domain difference but with the risk of deviating too much such that the translation of E(x,) + & no longer corresponds to y4. Thus, achieving balance is essential by choosing optimal A. From the results, we observe that À = 1.5 is optimal.
Methods
En-Fr
En-Cs
DARTS
38.42
25.00
No-MMDDdartts
37.70
24.30
38.8
Le-darts
38.64
25.04
No-Stage-II-darts
38.66
25.09
38.6
Ours-darts
38.85
25.24
39.4
PDARTS
39.39
25.33
No-MMDD-pdrrts
37.85
25.09
38.2
La-pdarts
39.49
25.44
2.0 38.1
Oo-Stage-lllpdarte
39.53
25.49
1.5
Ours-pdarts
39.73
25.89
Figure 2:
BLEU score of Ours-
Table BLEU scores on the (En-Fr) and English-Czech (En-Cs) datascts, under various abla- tion study settings: 1) removing the MMD loss lerm (No-MMD), 2) replacing MMD loss with L2 loss (L2), and 3) removing the second stage from our framework (No-Stage-Il).
of WMT English-Frenn
darts on WMT (En-Fr) with varying X.
Impact of the MMD loss term. We study the impact of the MMD loss term in our framework and its effect on downstream performance. This experiment carried out by 1) omitting the MMD loss term (denoted as No-MMD) and 2) replacing the MMD loss term with an L2 regularization on (denoted as L2). The training data was WMT14 English-German The test data was WMT14 English- French and WMT18 English-Czech. Table shows the results.
We observe that No-MMDDdarts and No-MMD-pddarts underperform compared to Ours-darts and Ours- pdarts, respectively, due to the absence of the MMD loss term. This absence leads to a degencrate solution of 8 = 0, resulting in the generated additional data identical to the real training data D and increasing the risk overfitting as both model weights and architecture parameters are optimized on Ddr, This is evident from the inferior performance of No-MMD-darts and No-MMDDpdarts compared to DARTS and PDARTS, which learn model weights and architecture paraneters on disjoint splits of the training dataset to prevent overfitting. The results underscore the critical role of the MMD loss in creating significant. domain gap
10

---

Published in Transactions on Machine Learning Research (12/2024)
Methods
DARTS-ood
Ours-darts
En-Ig (Ubuntu)
2.20 + 0.05 2.39 + 0.07
Table 5: BLEU score comparison between DARTS-ood and Ours-darts, which use real OOD validation data and generated OOD data to optimize architectures respectively. BLEU scores are measured on the Ubuntu (En-Ig) dataset.
between the generated MT data and D which is crucial for improving the OOD generalization performance of searched architectures subsequently.
Furthermore, we observe that Ours-darts and Ours-pdarts outperform L2-darts and Læ-pdarts, respectively, on both language pairs. This indicates that MMD outperforms the L2 distance in quantifying and amplifying the differences between domains. The calculation of MMD involves all data examples from both datasets, thereby more accurately refiecting dataset differences at a distributional level. It achieves this by calculat- ing the kernel-based dissinilarity for each pair of data examples, both within and between datasets, then combining these dissimilarity measures to form comprehensive MMD score. In contrast, the L2 distance measures are limited to comparing pairs comprising real sentence and its perturbed counterpart, focusing solely on individual data example disparities rather than assessing the dataset level difference. On the other hand, L2-darts and L2-pdarts outperform vanilla. DARTS and PDARTS. Although 1.2 distance is not as effective as MMD, it can still enlarge the domain difference between generated data and real data, thereby improving architectures' OOD generalization performance.
Impact of removing Stage II. In this study, we explore the significance of Stage 1l within our framework by removing it (denoted as No-Stage-IT). Rathor than acquiring the perturbations through learning. we randomly generate Gaussian noise and employ this the perturbations. These are then added to the embeddings of real MT sentences to create the synthetic OOD data for neural architecture search. After removing Stage 11, the original levee- optimization framework is reduced to bi-level formulation:
mina EN (((((x,)) + + W (A),
(10)
s.t. W (A) = argminw Σ (f((((z)); (,, w)
where 8, is drawn from a Gaussian distribution with a mean 2.7689 x 10-5 and a standard deviation of 0.0026). These parameters were derived from the statistics of the learned 8 Ours-darts.
The results, presented in Table 4 reveal that both No-Stage-II-darts and No-Stage-I-pdarts exhibit de- creased performance compared to their counterparts Ours-darts and Ours-pdarts. This outcome strongly underscores the importance of Stage II, which learns perturbations instead of setting them randomly. In Ours-darts and Ours-pdarts, the perturbations are learned to create the 'optimal' synthetic OOD data that is best suitable for evaluating and improving the OOD generalization performance of architectures. In contrast, the generated approximated OOD data in No-Stage-Il-dart and Po-Stage-II-pdaane is randomly generated, which sub-optimal. Notably, No-Stage-II-darts and To-Stago-II-pdarrs still outperform vanilla DARTS and PDARTS, suggesting that even randomly generated perturbations create additional data can improve the OOD generalization performance of architectures, compared to not using OOD data at all.
Impact of generating out-of-domain validation data. We investigate the effect of generating an additional dataset to approximate OOD data for architecture search within framework, rather than relying pre-existing OOD datasets. For this ablation study, conducted experiments Igbo language pair. We used CCMatrix as the training set and Gnome as the OOD validation set to perform architecture search using DARTS. Ubuntu was used as the lest. set. The results, presented in Table 5 show that our method outperforms this ablation setting, highlighting the importance of generating synthetic OOD dataset within our framework. Compared with a fixed OOD dataset, the generated dataset can be more diverse since it is explicitly optimized to be OOD. This increased diversity can lead more robust OOD generalization performance.
the English-
11

---

Published in Transactions on Machine Learning Research (12/2024)
Self Attention
Self Attention
GLU
GLU
Identity
Idencity
Encoder Layer
Encoder Layer
Self Altention
Cross
Crose
Croes
Self Atlention
Croes attention
FFN
FFN
Decoder Layer
Decoder Layer
Ours-darts
Ours-pdarts
Figure Architectures searchod by Ours-darts and Ours-pdarts on the WMT (En-De) datasct. Each architecture consists of encoder layers and decoder layers. Multiple identical layers with searched architecture are stacked to construct the final model. Green boxes represent search nodes that optimized during the search phase, while yellow boxes indicate predefined layers which are fixed during the search process.
Methods
En-De
En-Fr
En-Cs
Ours-darts
27.84
38.85
25.24
Ours-darts,
28.79
41.64
26.44
Ours-pdarts
28.24
39.73
25.89
Ours-pdarts,
29.29
42.97
27.26
Table 6: BLEU score comparison between Ours-darts, Ours-pdarts and their larger architecture versions Ours-darts and Ours-pdartsy, across En-De, En-Fr, and En-Cs language pairs, demonstrating the benefits of model scaling.
Impact of model size. To investigate the impact of final model size on machine translation performance, we conducted search to develop larger architectures, denoted as Ours-darts and Ours-pdartsy. Following the "Transformer-big architecture design Vaswani al. (2017). our large model architecture consists of 6 encoder and decoder layers, hidden size of 1024, a filter size of 4096, and 16 attention heads. Evaluations were conducted across three language pairs: English-German, English-Frenchh and English-Czedh in Table The results demonstrate that increasing the final model size significantly improves translation performance on these language pairs, underscoring the importance of model scaling in achieving higher translation accuracy.
detailed
5 Discussion
In-domain generalization performance. The superior in-domain generalization performance of our methods can be attributed to several interrelated factors. Firstly, synthesizing approximated OOD data during the architecture search phase exposes the model to wider array of linguistic features, enhancing its ability to generalize across diverse scenarios. This exposure likely leads to the selection of more robust neural architectures that are inherently better at handling not only OOD but also in-domain data. Additionally, training with OOD data may serve as form of regularization, preventing overfitting and promoting a more generalized understanding of the language, which in turn facilitates in-domain generalization.
Model size and computational costs We compare the total number of parameters for method and all baselines in Table 7 Ours-darts has fewer parameters than DARTS and Transformer. Ours-pdarts has fewer parameters than PDARTS. This indicates that our method significantly improves the OOD MT performance without increasing model size. The computational costs of conducting architecture search with Ours-darts, DARTS, Ours-pdarts, and PDARTS are presented in Table S in terms GPU days. Our mothods, including Ours-darts and Ours-pdarts, do not incur significantly higher costs (less than 10%) compared to baseline methods. This marginal increase in computational expense is notably outweighed by the significant performance enhancements observed across various OOD machine translation tasks.
12

---

Published in Transactions on Machine Learning Research (12/2024)
Methods
Transformer
DARTS
Ours-darts
PDARTS
Ours-pdarts
Paramcters
60.8M
62.8M
54.5M
73.4M
73.4M
Table Number of model parameters.
Methods
DARTS
Ours-darts
PDARTS
Ours-pdarts
Cost
3.13
3.40
9.26
9.59
Table 8: Comparison of search costs in GPU days for baseline methods DARTS, PDARTS, and our met Ours-darts and Ours-pdarts.
Analysis of searched architectures We analyze the architectures searched by our methods, shown in Figure g with green boxes indicating the search nodes optimized during the search phase, and yellow boxes representing nodes with predefined architectures. The architecture learned by Ours-darts utilizes self-attention for input processing and GLUs for selective feature enhancement in the encoder, while its decoder refines the output iteratively with multiple cross-attention layers. In contrast, the architecture searched by Ours-p sarssp which yields the best results, features streamlined encoder that employs self- attention for feature extraction. The extracted features are further maintained through identity layers. The decoder more complex, incorporating both self-atlention for refining its own output and cross-allention for integrating encoder information, further enriched by two FFNs for improved feature processing. Interestingly, the scarched architectures opt to exclude convolution operations, implying that the self-attention and cross- attention mechanisms adequately capture the relevant features for the MT task. This effectiveness is likely augmented by operations such as GLUs and FFNs, which improve feature representation without the need for convolutions. This observation in line with the architecture choices of popular Transformer models that also omit convolution operations Devlin et al. (2018); Liu al. (2019); Lewis al.) (2019); He et al.) (2000 Yang et al. (2019); Brown al. (2020). Further comparisons between the architectures searched by our method and those found by bascline methods are provided in Appendix
6 Conclusions and Future Works
In this paper, we propose novel multi-level optimization framework for searching O0D-generalizable Trans- former architectures for MT tasks. Our method automatically generates additional data from the available training data that approximates OOD data, which is subsequently used to enhance the OOD generalization performance of the scarched architectures. Our framework consists of three optimization stages performed end-Lo-end: 1) Training Transformer model weights on original real MT training data, 2) Generaling addi- tional synthetic OOD MT data that diverges domain from the real training dataset, 3) Searching for the Transformer architecture that minimizes loss on this generated approximated OOD MT data. We demon- strate the effectiveness of our method across a variety of OOD machine translation tasks. Furthermore, our searched architectures also achieve high performance on various in-domain MT tasks. Moreover, through various ablation studies, we further highlight the importance of learning adversarial perturbations through our formulation to generate additional synthetic OOD data for OOD generalizable Transformer architecture search.
In this work, our focus has been on applying our method to MT tasks, with plans to extend to a variety of NLP lasks such as texi classification, named entity recognition, and text summarization. Additionally, we are exploring its applicability other modalities, including image and audio data. This broadened scope aims to enhance method's versatility and robustness in addressing OOD generalizable architecture scarch across diverse lasks and modalities.
7 Acknowledgement
This research was supported by NSF TIS2405974 and NSF IIS2339216.
13

---

Published in Transactions on Machine Learning Research (12/2024)
References
Orevaoghene Ahia, Julia Kreutzer, and Sara Hooker. The low-resource double bind: An empirical study of pruning for low-resource machine translation. In Conference on Empirical Methods in Natural Language Processing, 2021. CRL Ctttttpsss//pp...eerticccoaoo ott.e/oa.....aao/ooorraaa::12111
Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. ArXio, abs/ 0os/1607.06440, 2016.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.00 1000:1444004444 0470,,4433...0 2014.
Fan Bao, Guoqiang Wu, Chongxuan Li, Jun Zhu, and Bo Zhang. Stability and generalization of bilevel programming in hyperparameter optimization. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.). Advances in Newral Information. Processing Systems, 2021. URL httpp:://pperreviev.
net/forum? -d
Atilim Gunes Baydin, Robert Cornish, David MartinezRRubio Mark Schmidt, and Frank D. Wood. Online learning rate adaptation with hypergradient descent. CoRR, abs/1703.04782, 2017. URL http::/arxii.
org/ hogg/bbs1703..c 04782
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-
Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Fric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Lan- guage models few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Newral Information Processing Systems, volume 33, pp. 1877-1901 Cur- ran Associates, Inc., 2020. URL /tttps:/pproceseee www.eediiinnnnnnnnnn.eeeeeeeeeee neurips.cc/paper httpp/ccaapaalelleepppppeeeaeeee /tlles//apperr2200/1lle (B37//0066bbc44444111688a3 http:o44110bb0011122444444 14ttreacc1264aaaaPPaaa pdf
Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search on target task and hardware. In ICLR, 2019.
Fabio Maria Carlucci, Antonio D'Innocente, Silvia Bueci, Barbara Caputo, and Tatiana Tommasi. Domain generalization by solving jigsaw puzzles. In CVPR, 2019.
Xin Chen, Lingxi Xie, Jun Wu, and Qi Tian. Progressive differentiable architecture search: Bridging the depth gap between search and evaluation. 2019 IEEE/CVF International Conference on Compuler Vision (ICCV), pp. 1294-1303, 2019.
Sang Keun Choe, Willie Neiswanger, Pengtao Xie, and Eric Xing. Betty: An automatic differentiation library for multilevel optimization. arXiv preprint arXiv:2207.02842 2022.
Sang Keun Choc, Sanket Vaibhav Mchta, Hwijeen Ahn, Willie Neiswanger, Pengtao Xie, Emma Strubell, and Eric Xing. Making scalable meta learning practical. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL httppp///pppeeeeeeeeeeeee//n/n// .eat:foorum.dde.ainn.......
Jacob Devlin, Ming-Wee Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.00g 04805, 2018.
Uí Dhonnchadha, Caoilhionn Nic Pháidín, and Josef Van Genabith. Design, implementation and evalua- tion of an inflectional morphology finite state transducer for irish. Machine Trunslation, 18(3):173 193, September 2003. ISSN 0922-6567. doi: 10. 10.1007/s105500000 2480-99. URL 510590-004-2480-1
Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. In International Conference on Machine Learning, 2017. URL https:://p/s semanticscholar.
org/CorrpusD::6636155
14

---

Published in Transactions on Machine Learning Research (12/2024)
Bonaventure F. P. Dossou and Chris C. Emezue. Okwugbé: End-to-end speech recognition for fon and igbo. ArXio, abs/2103.07762, 2021. URL
semanticschollar org/CorpusIDD httpa/acttccccboooooooo///r/appa 232233062
Matthias Feurer, Jost Springenberg, and Frank Hutter. Initializing bayesian hyperparameter optimization via meta-learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1126 1135. JMLR. org, 2017.
Luca Franceschi, Michele Donini, Paolo Frasconi, and Massimiliano Pontil. Forward and reverse gradient- based hyperparameter optimization. In International Conference on Machine Learning, pp. 1165-1173. PMLR, 2017.
Luca Franceschi, Paolo Frasconi, Saverio Salzo, Riccardo Grazzi, and Massimiliano Pontil. Bilevel pro- gramming for hyperparameter optimization and meta- leatalleanning. In International Conference on Machine Learning, pp. 1568-1577, PMLR, 2018.
Bhamu Garg, Li Zhang, Pradyumna Sridhara, Ramtin Hosseini, Eric Xing, and Pengtao Xic. Learning from mistakes a framework for neural architecture search. Proceedings of the AAAT Conference on Artificial Intelligence, 2022.
Muhammad Ghifary, W. Kleijn, Mengjic Zhang, and David Balduzzi. Domain generalization for object recog- nition with multi-task autoencoders. 2015 IEEE International Conference on Computer Vision (ICCV), pp. 2551-2559, 2015.
Riccardo Grazzi, Luca Franceschi, Massimiliano Pontil, and Saverio Salzo. On the iteration complexity of hypergradient computation. Tn International Conference on Machine Learning, pp. 3748-3756 PMT.R, 2020.
Arthur Gretton, Karsten M. Borgwardt, Malte Rasch, Bernhard Scholkopf, and Alex Smola. A kernel two-sample test. J. Mach. Learn. Res., 13:723 773, 2012. URL https::/api..seaanii sew.aiicccc....cccccccccc....... org/
10774520:10742222
Thomas Grubinger, Adriana Birlutiu, Holger Schöner, Thomas Natschläger, and Tom M. Heskes. Multi- domain transfer component analysis for domain generalization. Neural Processing Letters, 46:845 855, 2017.
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition supple- mentary materials. 2016.
Pengeheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced bert with disentangled attention. arXio preprint arxiv 200600666. 2020.
Xuehai He, Zhuo Cai, Wenlan Wei, Yichen Zhang, Lunt Mou, Eric Xing, and Penglao Xie. Towards visual question answering on pathology images. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Pupers), pp. 708 718, 2021.
Sepp Hochreiter and Jürgen Schmidluber. Long short-term memory. Neural computation, 98)5 1997.
Mingyi Hong, Hoi-To Wai, Zhaoran Wang, and Zhuoran Yang. A two-timescale framework for bilevel optimization: Complexity analysis and application to actor-critic. arXiv preprint arXt::2007.05170 2020.
Neel Jain, Ping yeh Chiang, Yuxin Wen, John Kirchenbauer, Hong-Min Chu, Gowthami Somepalli, Brian R. Bartoldson, Bhavya Kailkhura, Avi Schwarzschild, Aniruddha Saha, Micah Goldblum, Jonas Geiping, and Tom Goldstein. NEFTune: Noisy emboddings improve instruction finetuning. In The Twelfth International Conference Learning Represendations, 2024. URL http::///pppaereeeeeeeeerrrrbbbp
15

---

Published in Transactions on Machine Learning Research (12/2024)
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arxiv:1412. 6980, 2014.
arXiv preprint
Vijay Konda and John Tsitsiklis. Actor-critic algorithms. Advances in neural information processing systems, 12, 1999.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequeeee pre-training for natural language generation, translation, and comprehension. arXiv preprint 0rXiv:1910.13461 2019.
Hao Li, Zheng Xu, Gavin Taylor, and Tom Goldstein. Visualizing the loss landscape neural nets. ArXiv, abs/1712.09913, 2017. URL. https://api. www.aatteeccooooooorroooooooeoee 3693333
Hanxiao Liu, Karen Simonyan, Oriol Vinyals, Chrisantha Fernando, and Koray Kavukeuoglu. Hierarchical 2017.
representations for efficient architecture search. arXiv preprint
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts:
Diffcrentiable architecture search. ArXiu,
abs/1806.09085 09055, 2018.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907: 11692, 2019.
Jonathan Lorraine, Paul Vicol, and David Duvenaud. Optimizing millions hyperparameters by implicit differentiation. In International Conference on Artificial Intelligence and Statistics, pp. 1540-1552. PMLR, 2020.
Dougal Maclaurin, David Duvenaud, and Ryan Adams. Gradient-based hyperparameter optimization through reversible learning. In International conference on machine learning, pp. 2113 2122. PMLR, 2015.
Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via invariant fea- ture representation. In Proceedings of the 30th International Conference International Conference Machine Learning Volume 28, ICML'13, pp. 1- 1-1.-118.-11 JMLR.org, 2013.
Troro Orife. Attentive sequence-to-sxxquencc learning for diacritic restoration of yorùbá language text. Interspeech, 2018. URL https: www.ssiccccccncnccac..a.aaa.aaaa org/CorpusID 4559436
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Annual Meeting of the Association for Computational Linguistics, 2002.
Hieu Pham, Melody Y. Guan, Barret Zoph, Quoe V. Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. In ICML, 2018.
Fengchun Qiao and Xi Peng. Uncertainty-guider model generalization unseen domains. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 6790-6800, June 2021.
Fengchun Qiao, Long Zhao, and Xi Peng. Learning to learn single domain generalization. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12556-12565, 2020.
Aniruddh Raghu, Jonathan Lorraine, Simon Kornblith, Matthew McDermott, and David K. Duvenaud. Meta-learning to improve pre-training. Advances in Neural Information Processing Systems, 34, 2021.
Aravind Rajeswaran, Chelsea Finn, Sham M Kakade, and Sergey Levine. Meta-learning with implicit gradients. Advances in neural information processing systems, 32, 2019.
Aravind Rajeswaran, Igor Mordatch, and Vikash Kumar. game theoretic framework for model based reinforcement learning. In International conference on machine learning, pp. 7953 7963. PMLR, 2020.
16

---

Published in Transactions on Machine Learning Research (12/2024)
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le. Regularized evolution for image classifier architecture search. In Proceedings of the anai conference artificial intelligence, volume 33, pp. 4780- 4789, 2019.
Zhongzheng Ren, Raymond Yeh, and Alexander Schwing. Not all unlabeled data are equal: Learning weight data in semi-supervised learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Bal- can, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 21786- 21797. Curran Associates, Inc., 2020. URL http:://pocccaaaa....eerrr..ocpp #4460/39a88225282441111111111111
pdf
Karsten Roth, Jac Myung Kim, A. Sophia Koepke, Oriol Vinyals, Cordelia Schmid, and Zeynop Akata.
Waflling around for performance: Visual classification with random words and broad concepts. 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 15700-15711, 2023. URL https:
Ryo Sato, Mirai Tanaka, and Akiko Takeda. A gradient method for multilevel optimization. Advances in Newral Information Processing Systems, 34, 2021.
Holger Schwenk, Guillaume Wenzek, Sergey Edunov, Edouard Grave, and Armand Joulin. Cematrix: Mining billions of high-quality parallel sentences on the web. In Annual Meeting of the Association for Computa- tional Lánguistics, 2019. URL https://api. semanticscholar wwwpechhhlarrooo/oooooooooooe OrggCorpusIDD 207863306
Amirreza Shaban, Ching-An Cheng, Nathan Hatch, and Byron Boots. Truncated back-propagation for bilevel optimization. In International Conference on Artificial Intelligence and Statistics, 2018. URL
Shiv Shankar, Vihari Piratla, Soumen Chakrabarti, Siddhartha Chaudhuri, Preethi Jyothi, and Sunita Sarawagi. Generalizing across domains via cross-gradient training. ICLR, abs/ 1874.80445.,,, 2018.
Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, and Deyu Meng. weih-weight- Learning an explicit mapping for sample weighting. In Advances in Nearal Information Processing Systems, 2019.
pp.
David R. So, Chen Liang, and Quoe V. Le. The evolved transformer. In International Conference on Machine Learning, 2019.
Sai Ashish Somayajula, Linfeng Song. and Pengtao Xie. A multi-level optimization framework for end-to-end text augmentation. Transactions of the Associalion for Compulational Linguistics, 10:343 358, 2022.
Sai Ashish Somayajula, Lifeng Jin, Linfeng Song, Haitao Mi, and Dong Yu. Bi-level finetuning with task- dependent similarity structure for low-resource training. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 8569-8588, 2023.
Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Nya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting, J. Mach. Learn. Res., 15:1929 1958, 2014.
Felipe Petroski Such, Aditya Rawal, Joel Lehman, Kenneth Stanley, and Jeffrey Clune. Generative teaching networks: Accelerating neural architecture search by learning to generate synthetic training data. In International Conference on Machine Learning, pp. 9206-9216. PMLR, 2020.
Joshua Tobin, Rachel Fong, Alex Ray, Jonas Schncider, Wojeiech Zaremba, and P. Abbeel. Domain ran- domization for transferring deep neural networks from simulation to the real world. 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 23-30, 2017.
Minh- Tuan Tran, Trung Le, Xuan-May Le, Mehrtash Harandi, Quan Hung Tran, and Dinh Q. Phung, Nayer: Noisy layer data generation for efficient and effective data-free knowledge distillation. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognation (CVPR), pp. 23860-2388 23869, 2023. URL https:
7/api.semante www.semicnnnchlar.....m. htt.sssh..a....oo/ooopppp//aaaa.
17

---

Published in Transactions on Machine Learning Research (12/2024)
Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In N7PS, 2017.
Riccardo Volpi, Hongseok Namkoong, Ozan Sener, John Duchi, Vittorio Murino, and Silvio Savarese. Gen- eralizing to unseen domains via adversarial data augmentation. In Proceedinggs of the 32nd International Conference on Newral Information Processing Systems, NIPS'18, pp. 5339-53499 Red Hook, NY, USA, 2018. Curran Associales Inc.
Alex Wang. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprinl Xiv: 07461, 2018.
Shujun Wang, Lequan Yu, Caizi Li, Chi-Wing Fu, and Pheng-Amn Heng. Learning from extrinsic and intrinsic supervisions for domain generalization. In European Conference on Computer Vision, 2020a.
Yulin Wang, Jiayi Guo, Shiji Song, and Gao Huang. Meta-semi: A meta-learning approach for semi- 02394
supervised learning. CoRR, abs/2007.02035 .23994, 2020b. URL
Pengtao Xie and Xuefeng Du. Performance-aware mutual knowledge distillation for improving neural archi- tecture scarch. CVPR, 2022.
Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. SNAS: stochastic neural architecture search. In ICLR, 2019.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R. Salakhutdinov, and Quoe V Le. Xlnet: Gen- eralized autoregressive pretraining for language understanding. Adoances in neural information processing systems, 32, 2019.
Li Zhang, Youwei Liang, Ruiyi Zhang, Amirhosein Javadi, and Pengtao Xie. Blo-sam: Bi-level optimization based finetuning of the segment anything model for overfitingg-reeeetinn semantic segmentation. In ICML, 2024. URL
Miao Zhang, Steven W Su, Shirui Pan, Xiaojun Chang, Ehsan M Abbasnejad, and Reza Haffari. idarts: Dif- ferentiable architecture search with stochastic implicit gradients. In International Conference Machine Learning, pp. 12557-12555 12566. PMLR, 2021.
Yuekai Zhao, Li Dong, Yelong Shen, Zhihua Zhang, Furu Wei, and Weizhu Chen. Memory-effiiient differen- tiable transformer architecture search. In Findings, 2021.
Guoqing Zheng, Ahmed Hassan Awadallah, and Susan T. Dumais. Meta label correction for learning with weak supervision. CoRR, abs/1991.03809, 2019. URL http:
Chen Zhu, Yu Cheng, Zhe Gan, Siqi Sun, Tom Goldstein, and Jingjing Liu. Freelb: Enhanced adversarial training for natural language understanding. In International Conference on Learning Representations, 2020. URL http:////pppaeeeeeeeeeee/rrrrara
Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.0153 2016.
Barrel Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoe Le. Learning transferable architectures for scalable image recognition. In CVPR, 2018
18

---

Published in Transactions on Machine Learning Research (12/2024)
A RL-based and EA-based Neural Architecture Search Methods
Early NAS methodologies Zoph & Le 20166: Pham et al. 2018 Zoph et al. 20188 are based reinforcement learning (RL). They train a policy network to generate optimal architectures by maximizing validation accuracy, which serves as the reward. These strategies are straightforward in principle and offer the flexibility to explore various search spaces. Nonetheless, they require substantial computational resources. Specifically, to evaluate the performance of a proposed architecture, it necessitales training on a dataset, a process that is notably eesource-intensvve and time-consuming.
Another category of NAS methods Liu ct al. 2017]; Real et al. (2019) employ evolutionary algorithms (EA). In these methods, archilectures are treated individuals within a population, each rated by fitness score that reflects its performance. Architectures that score higher are more likely to produce offspring (i.c., new architoctures), which then supplant those with lower fitness scores. Similar to RL-based methods, EA-based methods are also marked by high computational demands, as they require the training each architecture to evaluate its fitness score. Recently, F.A-based NAŠ methods have been applied to search for optimal architectures of Transformers So et al. (2019), and achieved state-of-the-art performance on multiple machine translation tasks.
B
Candidate Operation Set
Following So et al.
2019 Zhao et al. (2021), the candidate operation in our experiments includes:
Feed-Ford Forward Neural Network (FFN)
Self Attention: head
ily: no transformation applied to the input
Standard Convolution: w x 1 with w E {1,3)
Depth-wise Separable Convolution:
with w E 73,5,7,9,11)
Dynamic Convolution: w x with w E (,,7, 7,1, 15)
Gated Linear Unit (GLU)
Cross Attention: head 8, only available to decoder
C
Optimization Algorithm
We first provide detailed compulation of the gradient used in our optimization algorithm. In Eq. 9 the gradient is calculated as follows:
V VA(((E((x)) + &. A),)) A (B A).W)) VAW" (BL (A).))) (KS(E())))8:WW (A)a)) 06
(11)
aw"
04
where 9. denotes partial derivative.
V can be calculated as:
V V = - V (((((()))))))+))))))))) +6; W A.,
(12)
W' can be calculated as:
N V = - V. Σ
(((E())):
(13)
19

---

Published in Transactions on Machine Learning Research (12/2024)
CCMatrix
Gnome
Ubuntu
En-Ig
5.9M
3173
608
En-Ha
80.4K
998
219
Table 9: Number of sentences in the CCMatrix, Gnome, and Ubuntu datasets for the English-Igbo (En-lg) and English-Hausa (En-Ha) language pairs.
We now discuss key properties of our proposed optimization algorithm. Previous research has extensively analyzed the convergence properties of optimization algorithms for bi-level optimization (BLO) problems. For instance, Franceschi et al. 2018 cstablished sufficient conditions under which solutions of approximate BLO problems converge to those exact problems. Similarly, Shaban et al. (2018) provided sufficient condi- tions for the convergence of approximate gradients obtained via truncated back-propagation during iterative optimization. Further, convergence analyses for implicit gradient-based methods have been explored in works such as Grazzi et al. (020)) and Zhang et al. (2021) Experimental results in this paper demonstrate that our proposed algorithm consistently converges and achieves superior performance compared to baseline meth- ods. However, our algorithm diverges from traditional BLO frameworks due to its multi-level optimization (MLO) structure, introducing unique challenges and opportunities for future exploration of its convergence properties.
Although approximated optimization algorithms for MLO problems, including our method, are theoretically and empirically likely to converge stationary points, it remains unclear whether they achieve global optima rather than sub-optimal solutions. Nevertheless, prior research has demonstrated that sub-optimal solutions can still yield superior generalization performance (Dinh et al. 2017 Li et al. 2017), finding corroborated by our experimental results. Furthermore, existing studies suggest that MLO algorithms enhance generaliz- ability and mitigate overfitting, attributed to their multi-stage optimization process and the use of distinct data splits (Somayajula et al. 2022 Bao et al. 2021). Specifically, by optimizing different sets of parameters on distinct data subsets, MLO-based methods effectively mitigate overfitting to single dataset (Zhang
et al. 2024
D Datasets
The Gnome dataset comprises 187 language pairs derived from the translation of Gnome documentation
while the Ubuntu dataset includes 42 language pairs generated from the translation of Ubuntu OS localization
files [ Each sentence in the Gnome and Ubuntu datasets has average length of 6-9 tokens. Table g
summarizes the dataset statistics. WMT-Chat dataset includes data for translating conversational text, in
particular customer support chats WMT-Biomedhe from the biomedical domain 5 Flores dataset is a multilingual machine translation benchmark, which consists
dataset includes data for the translation of documents
of translations primarily from English into around 200 language varieties
Here, we further discuss the discrepancies among the CCMatrix, Gnome, and Ubuntu datasets. Gnome contains technical and Ul-related text. specific to the Gnome environment, while Ubuntu contains system messages and interface text in Ubuntu OS, representing software domain. The text in this domain includes specific terms ("permissions", "kernel", "GUI", "repository", "conf "configuration file"), technical verbs ("execute," "compile," "deploy." ") and abbreviations ("sudo, " "bash," "root"). Sentences are typically task-oriented or informational, dealing with instructions, warnings, and descriptions of software functionality.
In contrast, CCMatrix is derived from multilingual web pages which cover a wide range of topics, representing general domain. For text in the general domain, it includes topics such politics, sports, entertainment,
hhttps://svx..
https://ubuntu.c
https://amt-hhat-aaaa github.bibbb..... lo/
https://www2.stattet wttp:/ttttt...t//m/////ie/eeeeee
20

---

Published in Transactions on Machine Learning Research (12/2024)
and daily conversations. The word usage is diverse, including those related to politics (oariicsscaarr ("democracy"), culture ("art gallery"), and emotions ("love"), which would be uncommon in software datasets.
To sum up, the difference between CCMatrix and Guome/Ubuntu datasets differs from multiple aspects including vocabulary and style, making them ideal datasets to investigate out-of-domain generalization performance of machine translation models.
E Baseline Methods
In the DARTS framework Liu et al. (2018), each search layer comprises multiple search nodes and the objective to determine the most suitable operation for each node from a predefined candidate set, denoted O. This set includes operations such FFNs, self-attention mechanisms, and so on (see Appendix B Each operation is applied to either the input of the layer or the outputs from intermediate nodes, generating outputs for subsequent processing. DARTS represents the output of each search node as weighted sum of outputs of all operations in O. The weights for each operation in this sum, pertinent to a search node, are derived from the softmax learnable parameters, referred to as architecture weights A. They act as selection variables, representing the likelihood of this operation being selected into the final architecture. Specifically, for the output of a search node f(-) and input 2, the operation defined as,
exp(a.) f(x) Σ OEO Eo'co exp(ax)) o(x)
(14)
where O is the candidate operation set, E O is an operation within this candidate set and {ao} are the architecture weights for the search node. DARTS utilizes a bi-level optimization framework to learn architecture weights A and model parameters W on two disjoint splits Dr and D2 of the training dataset. W is optimized by minimizing a loss L on data split D2 in the lower level, and A is learned on data split D2 in the upper level:
min L(W (A), D2) s.t. W* (*) aargmnn L(W,A,
(15)
(16)
This optimization problem is solved using one-step gradient descent and finie-ddiferrence approximation
similar to the one described in Section 3.6 After convergence, the final architecture is determined by selecting the operation with the highest architecture weight for each search node. For computational and memory efficiency, DARTS initially searches for the architecture with only a few search layers. After the searching phase, there a model retraining phase: a larger model (called final model) with more layers is composed by stacking the searched layer multiple times; then the final model retrained on the combination of data splits D, and D2- However, this approach introduces a discrepancy between the search phrase and retraining phrase, in terms of the number of layers in the models.
PDARTS Chen et al. 2019) addresses this limitation by progressively increasing the architecture's depth during the search phase. Initially, PDARTS trains the architecture with a few layers for several epochs, then refines the operation set to only include those with high architecture weights. This procedure is repeated, gradually increasing the number of layers to match the final model's layer number. However, this approach leads increased computational costs.
F Further Analysis of Searched Architectures
In this section, we present the architecture searched by baseline method, DARTS, as shown in Figure 4 for further comparison with the architecture scarched by our method, Ours-darts. We observe that both DARTS and Ours-darts result in cross attention modules, highlighting their effectiveness in the machine translation task. However, DARTS method leads to multiple feed-forward network (FFN) layers, which are absent. in
21

---

Published in Transactions on Machine Learning Research (12/2024)
Self Attention
Self
FFN
Attention
Encoder Layer
Sclf Atsention
Cross Atlention
Cross Attantion
FFN
Decoder Layer
Darts
Figure 4: Architectures searched by darts the WMT (En-De) dataset. The architecture consists of encoder layers and decoder layers. Multiple identical layers with scarched architecture are stacked to construct the final model. Green boxes represent search nodes that are optimized during the search phase, while yellow boxes indicate predefined layers which are fixed during the search process.
Histogram Angles Data
35
Frequency
20
15
5-
0
87.0
88.5
90.0
Local Angles
Figure 5: Distribution of local angles for generated approximated OOD samples.
architectuue searched from Ours-darts. Instead, our method discovers GLU modules for the encoder layer, which can dynamically adjust their memory unit based on the input, potentially offering better adaptability to domain shifts compared to FFN layers.
G Analysis of Generated Samples in Stage II
In this section, analyze some characters of the generated approximated OOD samples in the Stage II of our framework for improving the OOD performance of searched architecture. For each generated sample, we select 3 of the data samples in the real training datasct that are closest to this generated sample. These 3 samples determines a plane, and we compute the angle between the generated sample and this plane. The distribution of angles are presented in Figure 3 with an average degree of 89.66. Results show that the local angles between generated samples are close to 90 dogree, which is almost vertical to the approximate plane of the training data. This provide potential underlying reason for our method to achieve good out-of-domain generalization performance.
22

---

Published in Transactions on Machine Learning Research (12/2024)
En-Ig (Gnome)
En-Ig (Ubuntu)
No-NAS
2.65 + 0.04
2.15 + 0.05
Ours-darts
2.97 + 0.01
2.39 + 0.07
Table 10: BLEU scores on two out-of-domain En-Ig datasets under an ablation setting: removing neural architecture search and only training vanilla Transformer using synthetic out-of-domain data generated in Stage II.
H Impact of Neural Architecture Search
In this section, conduct additional experiments to investigate the impact of neural architecture search in our method. We introduce additional baseline by replacing the optimization of the trainable architecture in Stage III with optimizing model weights of a Transformer model instead (denoted as No-NAS). We present the results of this ablation study in Table I0 We observe that Ours-darts significantly outperforms the No- NAS baseline, highlighting the necessity of using neural architecture search. It also demonstrates that the out-of-domain samples generated in the second stage of our method do benefit the NAS process, instead of only being beneficial model weights in general deep learning process.
23


Reviews:
[{"review_id": "aXcHn5HdTO", "reviewer": "Reviewer_GTSQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nThe paper explores an interesting direction. Typically, for adapting to a new domain a pre-trained model is finetuned on domain specific data to modify the model parameters. This paper raises an interesting question of whether the architecture itself would be different for out-of-domain data?\n\n**Weaknesses** \n\nI will categorize the weaknesses broadly as conceptual weaknesses in the formulation of the solution and issues with the experimental setup. \n\n_Conceptual weaknesses_ \n\n- The idea behind Stage 2 which generates synthetic data is to create out of domain data which will be used to train Stage 3. However, I am not sure that the proposed method used to create synthetic data results out of domain data.  All that is being done is that noise is added to the input embeddings and Stage 2 ensures that the model generates the same original translations on the target side (which means no change in semantics).  With no change in semantics, how does the generated data quality as out-of-domain data. What do the authors mean by out of domain data? My understanding would be there would be terms not encountered in the training data, or terms which have new meanings in the new domain, etc. How does the proposed approach ensure this? \n- Adding noise with learned weights is probably some  smart regularization which might improve translation.\n- There is not even a target domain that is specified - can you show some examples of the kind of sentences created with the perturbation (I suppose that is non-trivial). \n- How do you justify the claim of “high-fidelity” OOD data – there is no evaluation of the quality of the synthetically generated data. \n- In Stage 3, the training is done exclusively on supposedly “out of domain data” - no general domain data is used as far as I could understand. How can synthetic data alone lead to good performance on the translation task? Typically, domain adaptation is achieved by a  mix of general domain and domain-specific data. Furthermore, while the weights of the models are learnt on general domain corpora (stage 1), the architecture is learnt on \"OOD\" corpora (stage 3) - that does not sound right to me. \n- Can you also mention how much synthetic data was created for Stage 3? \n- Is there a degradation in general domain performance for the final model?\n\n_Weaknesses in the experimental setup_ \n\n- Very low resource languages like Igbo and Hausa have been chosen for OOD evaluation. I am curious about the choice of these extremely low-resource languages to study OOD translation. It is difficult to get general domain performance at a decent level for these languages  - let alone performance in certain domains. I think this choice of OOD test data is not appropriate. \n- The GNOME and Ubuntu datasets themselves are not great test sets - they are pretty noisy and not appropriate for evaluation. \n- It would have been better to look at domain specific datasets in high resource languages The authors can take a look at the WMT shared tasks in many of the previous years for many domain specific datasets.  \n- **As the results show, the BLEU scores are very low (1-3) for OOD translation in low-resource languages. Those scores are basically meaningless and there is no translation happening – probably the source is just getting copied over.**\n- Table #2: shows results of training on one language pair and testing on two different language pairs (English French and English Czech).  This can hardly be considered an out of domain experiment. These are cross lingual zero-shot translation experiments which are way more challenging than OOD translation.  I am skeptical of how such high BLEU scores for English to French/Czech translation were obtained by just training on English-German corpora since these are totally different languages. Something is not correct here. \n- Table #4: It is not clear why the DARTS and PDARTS approach shows such degradation as compared to a vanilla transformer architecture – these approaches are supposed to work well for at least for in-domain scenarios and hence this is quite surprising. The proposed approach on the other hand seems to do better for in-domain scenarios though it was designed for out of domain scenarios. These observations of course hold only for Igbo and Hausa and not for German. Can you explain these? Testsets could themselves be an issue. CCMatrix is not a very clean corpus to use that as a test set, particularly for low-resource languages. A better testset would be the Flores-200 collection.\n\nIn the light of these major limitations in the experimental setup, it is difficult to establish the efficacy of the proposed approach.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper the authors present an approach for neural architecture search to improve machine translation on “out-of-domain” distribution data by finding architectures better suited to OOD  translation. The proposed solution builds on the DARTS framework based on differentiable search techniques. The key proposal is adding an additional intermediate stage in the multi-objective optimization that generates out of domain data, which is used for optimizing the last stage for architecture search.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* The paper needs a thorough overhaul of the evaluation setup as mentioned in weaknesses to be able to get interpretable results.\n* Clarification of the conceptual weaknesses - how does perturbation result in out-of-domain data and samples of the same. \n* An experiment of finetuning a base model with \"OOD data\" to see if finetuning with this data is sufficient to improve translation quality (in a revised experimental setup as suggested above).", "broader_impact_concerns": "No broader impact concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "fMlg2xzYCk", "reviewer": "Reviewer_6KDA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I found this work to be very interesting, and the proposed optimization procedure is clever. It is especially interesting to target OOD generalization by learning to generate new OOD data. The general topic of OOD generalization in machine translation is also important, and the proposal seems to yield improved performance. \n\nI do have a couple of questions and concerns regarding this work. First, the 'OOD' test data is drawn from different translation data for the same languages (e.g. training on CCMatrix and testing on Gnome and Ubuntu), but in what sense is this data out-of-distribution? Are there are any systematic differences between the training and test data, or is this setting labelled 'OOD' simply because they are from different datasets?\n\nSecond, my primary concern with the work is that, although the optimization procedure is clever, and seems to show improvements, it is not clear that these improvements are really specific to architecture search. I think an important control condition would be to use the same optimization procedure, but rather than using differentiable architecture search in the third step, simply update the base model. Are there really any benefits to optimizing the model through architecture search, rather than through standard learning methods? It is currently unclear whether the gains come from the use of architecture search vs. the method for generating synthetic OOD data.\n\nFinally, I just want to clarify that the OOD test data is not in any way used during the optimization procedure, i.e. the only data used for architecture search is the synthetic OOD data, not the data that's ultimately used to evaluate OOD performance? (I believe this is the case, but want to confirm).\n\nMinor note: the second paragraph of the intro uses the phrase 'DARTS-based' but this acronym has not yet been introduced.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work presents a novel approach for architecture search with transformers that is aimed at improving OOD generalization in machine translation. A 3-stage optimization procedure is proposed, involving first training a base model, then generating OOD data by learning a perturbation in embedding space, and finally updating the architecture so as to improve performance on the OOD data. The approach shows improvements for transfer generalization between different machine translation datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "In what sense is the OOD test data out-of-distribution?\n\nCan a control experiment be performed in which architecture search is replaced with standard learning methods (but which preserves the use of synthetic OOD data)?\n\nIs the OOD test data in any way used during the architecture search procedure, or only used to evaluate the resulting architecture?", "broader_impact_concerns": "There are no broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "eXvhi6f3gR", "reviewer": "Reviewer_4pCy", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n•\tThe paper presents a novel method of incorporating perturbation into embeddings, resulting in substantial performance improvements.\n\n•\tThe authors demonstrate a well-optimized perturbation process using MMD, which further enhances overall performance.\n\n\nWeaknesses:\n\n•\tThe proposed method requires an additional search cost compared to baseline methods such as DARTS or PDARTS.\n\n•\tThe current evidence does not fully elucidate how the perturbation process directly impacts the architecture obtained by NAS. It is possible that the observed performance improvements stem more from the perturbation process enhancing general deep learning training rather than specifically benefiting the NAS procedure itself. This distinction warrants further investigation to fully understand the mechanism of improvement.\n\n•\tWhile the paper provides valuable insights, there may be room to expand the literature review. Including a wider range of references, especially from recent work, could further enrich the paper's positioning within the field and potentially uncover additional connections to related research.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a novel method which improves performance of existing methods in Neural Architecture Search (NAS), such as DARTS or PDARTS, by introducing a perturbation process into the multi-level optimization framework. The effectiveness of the proposed perturbation method in enhancing NAS performance is validated through comprehensive ablation studies. These studies demonstrate that optimizing the perturbation degree by maximizing the maximum mean discrepancy (MMD) is crucial for improving overall performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The authors are encouraged to include recent references, particularly those exploring noise embeddings, such as Neftune (2023), to provide a broader context and strengthen the related work section.\nAdditionally, it is recommended to provide a comparative analysis of the architectures generated by DARTS and the proposed method to better illustrate the effects of the proposed perturbation technique.\n\nReference\n- Jain, N., Chiang, P. Y., Wen, Y., Kirchenbauer, J., Chu, H. M., Somepalli, G., ... & Goldstein, T. (2023). Neftune: Noisy embeddings improve instruction finetuning. arXiv preprint arXiv:2310.05914.", "broader_impact_concerns": "The paper highlights the potential for performance improvement in NAS through appropriate perturbation methods.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "FqOklVwmGP", "reviewer": "Reviewer_yu1j", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- The experiments are convincing, methodological details are sufficient, and the results are therefore likely to be reproducible. \n-  The paper is well motivated and addressed an important problem.\n- The methodology is simple, yet elegant and significant. Thus, it can be presented to be accessible to a broader audience and can have further interesting applications. \n\n**Weaknesses**\n\n- I would have appreciated a proper discussion section, instead of sparse discussion points within results. This decreases the clarity of the writing.\n- The related work section relates minimally back to current contributions. I think the distinctions should be made more clearly and the reader should be guided better. It is too generic in its current form.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a multi-level optimization framework to search for transformer architectures, which is illustrated for machine translation. The main advance is the focus on the out of domain generalization, in which the authors generate artificial datasets and retrain and evaluate candidate models. The approach is evaluated on several datasets, showing superior OOD generalization performance, improving the state-of-the-art methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I believe this is a solid piece of work. I recommend accept as is and have minor requests:\n\n1) Please address the weaknesses.\n\n2) I am wondering about the angles that the ood samples have with the embedding manifold. Can you perhaps do one final study, in which you compute local angles between the manifold defined by the training data and the new added data? You can do so for example by picking (for each ood sample) the closest three real data points and computing an approximate plane, with respect to which you can compute ood data angles. I suspect ood data that have close to 90 degree angles will lead to better generalization, as they will be in orthogonal dimensions to existing data embeddings. Can you perhaps quantify this by using ood data for particular angles and showing us the effects? I think such an experiment can help you argue for the underlying reason for why this approach works and can lead to generalizable insights.\n\nEdit: After reading other reviews, I sustain my belief that this work satisfies TMLR's criteria for acceptance.", "broader_impact_concerns": "Somewhat, please add a discussion section.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Shared above.
Decision: Accept (poster)