Example-based Hypernetworks for Multi-source Adaptation to Unseen Domains
Authors: Tomer Volk
Eyal Ben-David
Ohad Amosy
Gal Chechik
Roi Reichart
Venue: EMNLP 2023 Findings (2023)
Keywords: Domain adaptation
our of distribution
cross-lingual
hypernetworks
prompting
TL;DR: We present an example-based Hypernetwork algorithm that facilitates adaptation from multiple source domains to previously unseen target domains.

Abstract: As Natural Language Processing (NLP) algorithms continually achieve new milestones, out-of-distribution generalization remains a significant challenge. This paper addresses the issue of multi-source adaptation for unfamiliar domains: We leverage labeled data from multiple source domains to generalize to unknown target domains at training. Our innovative framework employs example-based Hypernetwork adaptation: a T5 encoder-decoder initially generates a unique signature from an input example, embedding it within the source domains' semantic space. This signature is subsequently utilized by a Hypernetwork to generate the task classifier's weights. In an advanced version, the signature also enriches the input example's representation. We evaluated our method across two tasks—sentiment classification and natural language inference—in 29 adaptation scenarios, where it outpaced established algorithms. We also compare our finetuned architecture to few-shot GPT-3, demonstrating its effectiveness in essential use cases. To the best of our knowledge, this marks the first application of Hypernetworks to the adaptation for unknown domains.

Example-based Hypernetworks for Multi-source Adaptation to Unseen Domains
Tomer Volk*1, Eyal Ben-David"', Ohad Amosy?, Gal Chechik Roi Reichart'
FFauulty of Data and Decision Sciences, Technion, IIT Bar Ilnn University, Israel
3 NVIDIA Research
(tomervolk heaaldd12allllllll roiri wew.orrriieteceiioonnmmmm ac. il (amosyoh gal. chechikk@biu.co
Abstract
To alleviate this difficulty, several OOD general- ization approaches proposed to use unlabeled data from the target distribution. For example, a promi- nent domain adaptation (DA, (Daumé III, 2007; Ben-David et al., 2010)) setting is unsupervised domain adaptation (UDA, (Ramponi and Plank, 2020)), where algorithms use labeled data from the source domain and unlabeled data from both the source and the target domains (Blitzer et al., 2006, 2007; Ziser and Reichart, 2017). In many real-world scenarios, however, expect training-tine access to target domain data. This could happen, for example, when the target domain is unknown, when collecting data from the target domain is impractical or when the data from the target domain is confidential (c.g. in healtheare applications). In order to address this setting, three approaches were proposed.
As Natural Language Processing (NLP) algo- rithms continually achieve new milestones, out- of-distribution generalization remains a signifi- cant challenge. This paper addresses the issue of multi-source adaptation for unfamiliar do- mains: We leverage labeled data from multiple source domains to generalize to unknown target domains at training. Our innovative framework employs example-hased Hyperne/work adapta- tion: T5 encoder-decodde initially generates a unique signature from an imput example, em- bedding it within the source domains' semantic space. This signature is subsequently utilized by a Hypernetwork to generate the task clas- sifier's 's weights. In advanced version, the signature also enriches the input example's rep- resentation. We evaluated method across two senkime classification and natural language inference-inn 29 adaptation scenar- ios, where it outpaced established algorithms. We also compare our finetuned architecture to effective- few-shot GPT-3, demonstrating ness in essential use cases. To our knowledge, this marks the first application of Hypemnt- works to the adaptation for unknown domains'.
impractical to
The first approach follows the idea of domain robustness, generalizing to unknown domains through optimization methods which favor robust- ness over specification (Hu et al., 2018; Oren et al., 2019; Sagawa et al., 2020; Wald et al., 2021; Zhao et al., 2020). Particularly, these approaches train the model to focus on domain-invarian features and overlook properties that are associated only with some specific source domains. In contrast, the second approach implements a domain expert for each source domain, hence keeping knowl- edge each domain separately. In this mixture- of-experts (MoE) approach (Kim et al., 2017; Guo et al., 2018; Wright and Augenstein, 2020; Man- sour et al., 2008), an expert is trained for each domain separately, and the predictions of these ex- perts are aggregated through averaging voting.
1 Introduction
Deep neural networks (DNNs) have substantially improved natural language processing (NLP), reaching task performance levels that were con- sidered beyond imagination until recently (Con- neau and Lample, 2019; Brown al., 2020). How- ever, this unprecedented performance typically de- pends on the assumption that the test data is drawn from the same underlying distribution as the train- ing data. Unfortunately, as text may stem from many origins, this assumption is often not met in practice. In such cases, the model faces an out-of- distribution (OOD) generalization scenario, which often yields significant performance degradation.
To bridge the gap between these opposing ap- proaches, third intermediate approach has been re- cently proposed by Ben-David et al. (2022). Their PADA algorithm, standing for a Prompt-based Au- toregressive Approach for Adaptation to Unseen Domains, utilizes
Both authors equally contributed to this work. 'Our code and data are available at https: thub. com/T htt/T//eeeooo//yprreeee
and domain-

---

Premise. Homes not located on one of these roads must place a mail receptacle along route traveled. Hypothesis. Other roads are far too raral to provide mail service to. Domain, Government. DRF Signature. travel: city, area, town, reports, mod- em
the first apply HNs to DA scenarios where la- beled data from the target domain, and actually also any other information about potential future test domains, are not within reach. Hence, we are the first to demonstrate that HNs generalize well to unseen domains.
Label. Entailment.
Fiction: jon, tommy, tuppence, daan, said, looked newsweek, reports, according, robert Telephone: yeah, know, well, really, think, something Travel: century, city, island, modern, town, built, area
Slate:
We propose three models of increasing complex- ity. Our basic model is Hyper-DN, which explicitly models the shared and Nomainn-specion aspects of the training domains. Particularly, it trains the HN on training data from all source domains, to gener- ate classifier weights in a domain-specific manner. The next model, Hyper-DRF, an example-based HN, performs parameter sharing at both the domain and the example levels. Particularly, it first gener- ates an example-based signature as in PADA, and then uses this signature as input to the HN so that it can generate example-specific classifier weights.? Finally, our most advanced model is Hyper-PADA which, like Hyper-DRF, performs parameter shar- ing at both the example and domain levels, using the above signature mechanism. Hyper-PADA, however, does that at both the task classification and the input representation levels. For detailed description see $3.
Table I: An example of Hyper- DRF and Hyper-PADA application to an MNLI example. In this setup the source training domains are Fiction, Slate, Telephone and Travel and the unknown target domain is Govern- ment. The top part presents the example and the DRF signature generated by the models. The bottom-part presents the DRF set of each source domain.
specific features to perform example-base based adap- tation. Particularly, given a test example it gen- erates a unique prompt that maps this example to the semantic space of the source domains of the model, and then conditions the task prediction on this prompt. In PADA, T5-based algorithm (Raf- fel et al., 2020), the prompt-generation and task prediction components are jointly trained on the source domains available to the model.
Despite their promising performance, none of the previous models proposed for DA in NLP ex- plicitly learns both shared and domain-specific as- pects of the data, and effectively applics them to- gether. Particularly, robustness methods focus only on shared properties, MoE methods train a separate learner for each domain, and PADA trains a single model using the training data from all the source domains, and applies the prompting mechanism in order to exploit example-specific properties. This paper hence focuses on improving generalization to unseen domains by explicitly modeling the shared and domain-speciff aspects the input.
We follow Ben-David et al. (2022) and exper- iment in the any-domain adaptation setup ($4.5). Concretely, we leverage labeled datasets from mul- tiple domains, excluding one for testing in our leave-onc-out experiments. Although designed for cross-domain (CD) generalization, we can ex- plore our models tross-language cross-domain (CLCD) setups using multilingual pre-trained lan- guage model. In CLCD and CD sentiment classifi- cation (12 settings each) and CD MNLI (5 settings), Hyper-PADA outperforms an off-the-shelf SOTA model (a fine-tuned T5-based classifier, without domain adaptation) by 9.5%, 8.4%. and 11.8% on average, respectively. Also, our results show that our HN-based methods surpass previous models from the three families described above. Lastly, ad- ditional comparisons show the value of individual components our HN-based algorithms, and rein- force the need for DA methods in the era of huge LMs when comparing Alyper-PADA to GPT-3.
To facilitate effective parameter sharing between domains and examples, we propose a modeling ap- proach based on Hypernetworks (HNs, Ha et al. (2017)). HNs are networks that generate the weights of another target network, that performs the learning task. The input to the HN defines the way information is shared between training examples. Mahabadi et al. (2021) previously focused on a sim- pler DA challenge, applying HNs to supervised DA, when small number of labeled examples from the target are used throughout the training procedure. Nevertheless, to the best of our knowledge, we are
DDFFs stand for Domain Related Features and DN stands for Domain Name. See SA..

---

2 Related Work
$ Positive
2.1 Unsupervised Domain Adaptation
Or
g
Most recent DA rescarch addresses UDA (Blitzer et al., 2006; Reichart and Rappoport, 2007; Glorot al.. 2011). Since the rise of DNNs, the main focus of UDA research shifted to representation learning methods (Titov, 2011; Glorot et al., 2011; Ganin and Lempitsky, 2015; Ziser and Reichart, 2017, 2018, 2019; Rotman and Reichart, 2019;
"Movies"
p "best film ever"
Figure 1: A discriminative model, based on hypernet- works. The HN (f), that is conditioned on the example domain (!), generates the weights (0,) for classifier (g), which is based on feedforward network.
Han and Eisenstein, 2019; Ben-David et al., 2020; Lekhtman al., 2021).
the domain name, generates the weights for the discriminative architecture (g). which, in turn, pre- diets the (positive) sentiment of the input review (p). HNs are formulated by the following equations:
Our paper considers recent DA setup, presum- ing no training-time knowledge of the target do- main, termed as any-domain adaptation (ADA) by Ben-David et al. (2022)). As discussed in $1, some papers that addressed this setup follow the domain robustness path (Arjovsky et al., 2019). while others learn a mixture of domain experts (Wright and Augenstein, 2020). Ben-David et al. (2022) presented PADA, algorithm trained data from multiple domains and adapted to test amples from unknown domains through prompting, PADA leverages domain related features (DRFs) to implement an example-base prompting mech- anism. The DRFs provide semantic signatures for the source domains, representing the similarities among them and their unique properties.
0, = f(1,0g) 89 - g(p 01)
(1)
(2)
Where f is the HN, g is the main task network, fy are the learned parameters of f. I is the input of f, and p is the representation of the input example. 0,, the parameters of network g, are generated by the HN f. and sy are the (task-specific) model pre- dictions. The classification loss and it's gradients are then calculated with respect to the HN. Ap- plied widely across computer vision (Klein et al., 2015; Riegler et al., 2015; Klocek et al., 2019), continual learning (von Oswald et al., 2020), feder- ated learning (Shamsian et al., 2021), weight prun- ing (Liu et al., 2019), Bayesian neural networks (Krueger et al., 2017: Ukai et al., 2018; Pawlowski et al., 2017; Deutsch al., 2019), multi-task learn- ing (Shen et al., 2018; Klocek et al., 2019; Serrà et al., 2019; Meyerson and Miikkulainen, 2019), and block code decoding (Nachmani and Wolf, 2019), their application within NLP remains lim- ited.
PADA is trained in multitask fashion source domain examples. For each example, it is trained either generate DRF signature or classify the example, using the signature as a prompt. Infer- ence involves PADA generating a DRF for its input and classifying it using this signature. The addi- tion of an architecture component, HNs, necessi- tates a two-phase training process, outlined in $3. Unlike previous DA work in NLP (and specifi- cally PADA), perform adaptation through HNs which are trained to generate the weights of the task classifier in domain-based or example-based man- ner. This framework allows us to explicitly model domain-invariant and domain-specifie aspects of the data and perform example-bane -asee adaptation.
HNs are effective in language modeling (Suarez, 2017), cross-task cross-language adapta- tion (Bansal et al., 2020; Üstün et al., 2022), and machine translation (Platanios et al., 2018). Ad- ditionally, Üstün et al. (2020) and Mahabadi et al. (2021) leveraged HNs in Transformer architectures (Vaswani et al., 2017) for cross- lingull parsing and multi-task learning, generating adapter (Houlsby et al., 2019) weights while keeping pre-trained lan- guage model weights fixed. Mahabadi al. (2021) addressed the supervised DA setup, where labeled data from the target domain is available.
2.2 Hypernetworks
Hypemetworks (HNs) (Ha et al., 2017) are net- works trained to generate weights for other net- works, enabling diverse, input-conditioned person- alized models. =In Figure/ we present an HN- based sentiment classification model. The model receives a review that originates from the "Movies" domain and the HN (f), which is conditioned on
Our work applies HNs to generate task classifier weights, jointly training the HN and fine-tuning

---

language model. We also incorporate example- based adaptation per Ben-Davide al. (2022), a novel HN application within NLP, marking the first instance of HNs deployed in NLP in an example- based fashion. Lastly, we introduce a HN mecha- nism designed for adaptation to unseen domains.
ken goes to the HN. In this model, neither domain- name nor example-specific signature is generated.
Hyper-DRF We hypothesize that parameter shar- ing in the domain level may lead to suboptimal performance. For instance, the sentence pair of our running example is taken from the Government do- main but is also semantically related to the Travel domain. Thus, we present Hyper-DRF (Figure 3c), an example- based adaptation architecture, which makes use of domain-related features (DRFs) in ad- dition to the domain name. Importantly, this model may connect the input example to semantic aspects of multiple source domains.
3 Domain Adaptation with Hypernetworks
In this section, we present our HN-based model- ing framework for domain adaptation. We present three models in increased order of complexity: We start by generating parameters only for the task classifier in domain-based manner (Hyper-DN), proceed to example-based classifier parametriza- tion (Hyper-DRe) DRF) and, finally, introduce example- based parametrization at both the classifier and the text representation levels (Hyper-PADA)
Hyper-DRF is a multi-stage multi-task autore- gressive model, which first generates a DRF sig- nature for the input example, and then uses this signature as an input to the HN. The HN, in turn, generates the task classiiiiee (CLS) weights, but, un- like in Hyper-DN, these weights are example-based rather than domain-based. The model is comprised of the following components: (1) a T5 encoder- decoder model which generates the DRF signature of the input example in the first stage (Iravel: city, area, town, reports, modern in our rumning exam- ple); (2) (separate) T5 encoder to which the exam- ple fed in the second stage; and (3) a HN which is fed with the DRF signature, as generated in the first stage, and generates the weights of the task- classifier (CLS). This CLS is fed with the example representation, as generated by the T5 encoder of (2), to predict the task label.
Throughout this section we use the running ex- ample of Table 1. This is a Natural Language In- ference (NLI) example from one of our experimen- tal MNLI (Williams et al., 2018) setups. In this task, the model presented with two sentences, Premise and Hypothesis, and it should decide the relationship of the latter to the former: Entailment, Contradiction or Neutral (see $4).
$3.1 describes the model architectures and their training procedure. We refer the reader to Ap- pendix A. for more specific details of the DRF scheme, borrowed from Ben-David al. (2022). The DRF's are utilized to embed input examples in the semantic space of the source domains, hence supporting example-based classifier parametriza- tion and improved example representation.
Below discuss the training of this model in details. The general scheme is as follows: We first train the T5 encoder-decoder of the first stage ((I) above), and then jointly train the rest of the archi- tecture ((2) and (3) above), which is related to the second stage. For the first training stage we have to assign each input example a DRF signature. In $A.1 we provide the details of how, following Ben- David et al. (2022), the DRF sets of the source train- ing domains are constructed based on the source domain training corpora, and how a DRF signature is comprised for each training example in order to effectively train the DRF signature generator ((1) above). For now, it is sufficient to say that the DRF set for each source domain includes words strongly linked to that domain, and each example's DRF signature is sequence of these DRFs (words).
3.1 Models
Hyper Domain Name (Hyper-DN) Our base model (Figure 3b) unifies a pre-trained T5 language encoder, classifier (CLS), and hypernetwork (HN) that generates classifier weights. Hyper-DN feeds the domain name into the HN. As domain names are unknown during test-time inference, we employ an "UNK" token to represent all unknown domains. To familiarize the model with this token, we apply it to an proportion of training examples during training instead of the domain name. This architecture promotes parameter sharing across do- mains and optimizes weights for unknown domains the classifier level.
During inference, when introduced to an exam- ple from an unknown domain, Hyper-DRi gener- ates its DRF signature using its trained generator
In Table 1, the test example's premise and hy- pothesis enter the T5 encoder, while the "UNK" to-

---

(T'S encoder-decoder), This way, the signature of test example may consist of features from the DRF sets of one or more source domains, forming mix- ture of semantic properties these domains. In our running example, while the input sentence pair is from the unknown Government domain, the model generates a signature based on the Travel and Slate domains. Importantly, unlike in Hyper-DN, there is no need in "UNK" token as input the HN since the DRF signatures are example-based.
P(Y[))
DRF Signature
CLS
x
T5
HN
T5
Encoder Encoder-Decode Decoder
Encoder
Prompt
Model-generated Signature
X
(a) Generator
(b) Discriminator
Figure 2: Hyper-PADA training. The generative (T5 encoder-decoder) and discriminative (HN, T5 Ecnoder and CL.S) components are trained separately, using source domains examples.
Hyper-PADA While Hyper-DRF implements example-based adaptation, parameter-sharing is modeled (apart from the T5 model) only at the classifier level: The language representation (with the T5 encoder) is left untouched. Our final model, Hyper-PADA, casts the DRF-based signature gen- erated at the first stage of the model, both as a prompt concatenated to the input example before is fed to the T5 language encoder, and as an input to the HN.
when using T5 encoder-decoder for the first stage, and separate T5 encoder for the second stage. Moreover, since the output of the first stage discrete sequence of words), we cannot train all components jointly.
Hence, we train each stage of these models sepa- rately, as shown in Figure 2. First, the T5 encoder- decoder is trained to generate the example-based DRF signature. Then, the HN and the (separate) T5 encoder are trained jointly with the task objective.
Specifically, the architecture of Hyper-PADA (Figure 3d) identical to that of Hyper-DRE, At its first stage, which is identical to the first stage of Hyper-DRF, it employs a generative T5 encoder- decoder which learns to generate an example- specific DRF signature for each input example. Then, at its second stage, the DRF signature is used in two ways: (A) unlike in Hyper-DRI, it is concatenated to the input example as a prompt, and the concatenated example then fed into T5 en- coder, in order to create a new input representation (in Hyper-DRI the original example is fed into the T5 encoder); and (B) as in Hyper-DRF, it is fed to the HN which generates the task-classifier weights. Finally, the input representation constructed in (A) is fed into the classifier generated in (B) in order to yield the task label.
Experimental Setup
4.1 Tasks, Datasets, and Setups
While focus is on domain adaptation, the avail- ability of multilingual pre-trained language en- coders allows us to consider two setups: (1) Cross- domain transfer (CD); and (2) cross- pross-language cross- domain transfer (CLCD). We consider multi-source adaptation and experiment in lcave-one-out fash- ion: In every experiment we leave one domain (CD) or one domain/language pair (CLCD) out, and train the datasets that belong the other domains (CD) the datasets that belong both other domains and other languages (CLCD; nei- ther the target domain nor the target language are represented in the training set).²
Training While some aspects of the selected training protocols are based on development data experiments ($4), we discuss them here in order to provide a complete picture of our framework.
For Hyper-DN, we found it most effective to jointly train the HN and fine-tune the T5 encoder us- ing the task objective. As discussed above, Hyper- DRF and Hyper-PADA are multi-stage models, where the HN (in both models) and the T5 lan- guage encoder (in hyper-PADA only) utilize the DRF signature generated in the first stage by the T5 encoder-decoder. Our development data ex- periments demonstrated significant improvements
Data set sizes Despite the growing ability to collect massive datasets, obtaining large labeled datasets is still costly and labor-intensive. When addressing a new task with a limited annotation budget, the choice lies between focusing efforts on a single domain or distributing the effort across multiple domains, acquiring fewer examples from
'RIL of the datascts, implementation details, and hyper- parameter configurations are described in Appendix D.

---

P(Y[))
P(Y[))
P(Y[X))
P(YP))
W
W
CLS
W
CLS
CLS
x
x
CLS
X
HN
T5
DRF
HN
T5 Encoder
DRF Signature
HN
Encoder
T5
Encoder
T5
T5 Encoder- Decodde-eecdderr
Domain Name
Ercoder-Decader
Prompt
X
X
(a) T5-NoDA
(b) Hyper-DN
(c) Hyper-DRF
(d) Hyper-PADA
Figure 3: The four models representing the evolution of HN-based domain adaptation framework. From left to right: T5-NoDA is standard NLP model comprised of a pre-trained T5 encoder with classifier on top of it, both are fine-tuned with the downstream task objective. Hyper-DN employs an additional hypemettork (HN), which generates the classifier (CLS) weights given the domain name "UNK" specifier for examples from unknown domains). Hyper-DRR and Hyper-PADA are multi-stage multi-task models (first-stage inputs are in red, second stage imputs in black), comprised classifier (CLS). At the first stage, the T5 encoder-decoder is trained for example- baseee DRF signature generation ($A.1). At the second stage, the HN and the T5 encoder are jointly trained using the downstream task objective. In Hyper-PADA, the DRF signature of the first stage applied both for example representation and HN-based classifier parametrization, while in Hyper-DRl DRFe--DRIiA it is applied only for the latter purpose. In all HN-based models. our HN is a simple two-layer feed-forward NN (SD).
a separate T5 encoder, HN and task
each while maintaining a similar overall data size. This work explores the latter scenario. To follow the experimental setup presented in previous DA works (Guo al., 2018; Wright and Augenstein, 2020; Ben- Ben-David et al., 2022) and to perform realis- tic experiments, we hence adjust our multi-domain datasets. In each experiment, we downsample each domain to have several thousand (3K-10K) training examples (with a proportionate development set).
ples (see details in $B).
Cross-language Cross-domain (CLCD) and Mul- tilingual Cross-domain (CD) Transfer for Senti- ment Analysis We experiment with the task of sentiment classification, using the Websis-CLS-10 dataset (Prettenhofer and Stein, 2010), which con- sists of Amazon reviews from 4 languages (English (En), Deutsch (De), French (Fr), and Japanese (Jp)) and 3 product domains (Books (B), DVDs (D), and Music (M)).
Cross-domain Transfer (CD) for Natural Lan- guage Inference We experiment with the MNLI dataset (Williams et al., 2018). In this task, each example consists of premise-hypothesis sentence pair and the relation between the the latter and the former: Entailment, contradiction, or neutral. The corpus consists of ten domains, five of which are split to train, validation, and test sets, while the other five do not have training sets. We experiment with the former five: Fiction (F), Government (G), Slate (S), Telephone (TL), and Travel (TR).
We perform one set of multilingual cross-domain (CD) generalization experiments and one set of cross-language cross-domain (CLCD) experiments. In the former, keep the training language fixed and generalize across domains, while in the latter we generalize across both languages and domains. Hence, experimenting in the CL.DD setting we focus each time on one domain/language pair. For instance, when the tar- get pair is English Boooks, we train on the training sets of the /French, Deutsch, Japanese) languages and the (Movies, Music/ domains (a total of 6 sets), and the test consists of English examples from the Books domain. Likewise, in the CD setting we keep the language fixed in each experiment, and generalize from two of the domains to the third one. We hence have 12 CLCD experiments (one with
leave-one-out fashion.
Since the MNLI test sets are not publicly avail- able, use the validation sets as test sets and split the training sets train and validation. Fol- lowing our above multi-domain setup, down- sample each domain so that in each experiment we have 10,000 training (from all source domains jointly), 800 validation and about 2000 test exam-

---

Deuisch
Eaglish
Fiench
Jigunes:
T5-NaDA
77.1
75.8
61.9
78.4
78.8
64.5
83.0
82.6
75.1
61.5
79.0
79.7
75.0
15-MoE-Ind-Avg
86.0
81.2
81.6
77.2
83.6
82.0
81.8
IS-Moo-Imn--Atin
76.2
79.6
86.0
82.6
81.7
84.6
77.4
81.8
82.2
82.9
T5-MoR-Aag
81.6
76.7
70.0
85.7
$1.5
81.6
85.0
84.8
77.0
82.2
83.4
81.9
81.7
TS.RRM
71.2
70.2
75.8
B0.8
72.5
73.0
8006
78.4
75.5
75.8
78.4
TS-DANN
82.1
77.8
80.8
84.5
78.8
79.0
84.2
82.6
77.2
68.7
78.8
81,6
79.7
PADA
57.7
74.8
74..2
71.8
15.9
75.8
81.8
82.0
76,8
77.2
75.8
80.0
75.8
Hypee-N
Hyper-DRy
K5.9
x1.2
34.6
86.4
84.01
K3.9
85.7
X5.5
81.4
K2.2
82.0
83.9
83.9
Rype-PADA
85,70
81,841
K5.0
86.01
84,44
85.1
M6.6
85.9
81,84
83,9
K3,9
K3.8
H4.5
Upper-bound
88.7
35.9
33.9
Table CLCD sentiment classification accuracy. The statistical significance of the Hyper-PADA results (with the McNemar paired test for labeling disagreements (Gillick and Cox, 1989), p 0.05) is denoted with: & (vs. the best of Hyper-DN and Hyper-DRF), + (vs. the best domain expert model), o (vs. the best domain robustness model), and $ (vs. PADA (example-basse baseee adaptation)).
each language/domain pair as target) and 12 CD experiments (for each language we perform one experiment with each domain as target). Follow- ing our above multi-domain setup, downsample each language-domain pair so that each experiment includes 3000 train, 600 validation and 2000 test examples (see details in ŞB).
Fine Time, and MoE Avg models presented by Wright and Augenstein (2020) For T5-MoE-Ind- Arg, we fine-tune expert model (with the same architecture as 75-NoDA) on the training data from each source domain. At inference, we average the class probabilities of all experts, and the class with the maximal probability selected.
For T5-MoE-Ind-Atth, we train an expert model for each source domain. Then, in order to find the optimal weighted expert combination, perform randomized grid search on our (source domain) development set. Finally, T5-MoE-Arg similar to T5-MoE-Ind-Aee except that we also include a general- domain expert, identical to T5-NoDA, in the expert committee.
4.2 Models and Baselines
We compare our HN based models (Hyper-DN, Hyper-DRF, and Hyper-PADA) to models from three families (see 1): (a) domain expert models that do not share information across domains: A model trained on the source domains and applied to the target domain with no adaptation effort (TS- NoDA); and three mixture of domainnspecifft ex- pert models (Wright and Augenstein, 2020), where a designated model trained on each source do- main, and test decisions are made through voting between the predictions of these models (TS-MoE- Ind-Avg, AaMo---nn-Amm and M5-MoE-Avg); (b) domain robustness models, targeting generaliza- tion to unknown distributions through objectives that favor robustness over specification (T5-DANN (Ganin and Lempitsky, 2015) and T5-IRM (Ar- jovsky et al., 2019)); and (c) example-based multi- source adaptation through prompt learning (PADA, the SOTA model for our setup).
(b.1) Bh-Invariant-Riske Risk- (T5- IRM). Using the same architecture as 75-NoDA, but with an objective term that penalizes repre- sentations with different optimal classifiers across domains.
(T5- DAN). An expert with the same architecture as T5-NoDA, but with additional adversarial do- main classifier head (fed by the T5 encoder) which facilitates domain invariant representations.
(b.2)
(c.1) PADA. A T5 encoder-dccoder that fed with each example and generates its DRF signature. The example is then appended with this signature as a prompt, fed again to the T5 encoder and the re- sulting representation is fed into the task classifier. We follow the implementation and training details from (Ben-David al., 2022).
Below we briefly discuss each of these models. All models, except from T5-MoE, are trained concatenation of the source domains training sets.
(a.1) 155-No-Domain--dappaaaia (T5-NoDA). A model consisting of a task classifier on top of T5 encoder. The entire architecture is fine-tuned on the downstream task (see Figure 3a).
For each setup we also report an upper-bound: The performance of the model trained on the train- ing sets from all source domains (or source lan- guage/domain pairs in CLCD) including that of
(a.2-4) T5-Mixture-of-Expeet (T5-MoE-Ind- Avg,
T5-MoE-Avg). Our im- plementation of the Independent Avg, Independent
"For the MoF models, we follow the naming conventions of Wright and Augenstein (2020).

---

the target, when applied to the target domain's 's (or language/domain pair in CLCD) test set.
D2
25
TP
A1
6.2
18.2
1.4
2.4
4.7
2.4
26.0
17.2
10.5
135
5 Results
T5MOE-Agg
1.2
4.7
0.1
63
-2.8
10.6
0
31.1
0.69
-03
0.6
10.4
23
754RM
0.6
3.5
29.0
28.5
1.7
31.9
-8.
Table 2 and Figure 4 present sentiment classifica- tion accuracy results for CLCD and CD transfer. respectively (12 settings each), while Table 3 re- ports Macro-F1 results for MNLI in 5 CD settings. We report accuracy Fl results for each setting, well the average performance across settings, Finally, we report statistical significance follow- ing the guidelines at Dror et al. (2018), comparing Hyper-PADA to the best performing model in each of the three baseline groups discussed in $4: (a) domain expert models (T5-NoDA and T5-MoE) :bb domain robustness models (T5-DANN and TS- IRM) and (c) example-based adaptation (PADA). We also report whether the improvement of Hyper- PADA over the simpler HN-based models, Hyper- DN and Hyper-DRF, significant.
1804
332.7
20.5
23.4
20.3
-22
1.7
3.4
31.
23.7
1
35.7
1.5
2.6
033
163
Hyper 1881
02
0.6
22.5
22.8
2.3
25.8
14
23.6
5
Figure 4: Accuracy improvements over T5-NoDA, in cross-domain (CD) generalization four languages: German, English, French, and Japanese. From the 28 out of 36 settings where Hyper-PADA outperforms the best model in each the baselines groups, in 23 cases the difference is significant (following Table 2 protocol).
mance of all baselines is within a 1% regime). This observation is related to another finding: Using the DRF-signature as a prompt in order to improve the example representation is more effective in CD MNLI - which is indicated both by the strong per- formance of PADA and the 3.1 F1 gap between Hyper-PADA and Hyper-DRF- than in CLCD and CD sentiment classification - which is indicated both by the weaker PADA performance and by the 0.6% (CLCD) and 1% (CD) accuracy gaps between Hyper-PADA and Hyper-DRI'
Our results clearly demonstrate the superiority of Hyper-PADA and the simpler HN-based models. Specifically, Plyper-PADA. outperforms all baseline models (i.e. models that do not involve hypernet- work modeling, denoted bellow as non-HN models) 11 of 12 CLCD settings, in of 12 CD sentiment settings, and in all 5 CD MNLI settings, with an av- erage improvement of 2.7%, 3.9% and 3.4% over the best performing baseline in each of the settings, respectively. Another impressive result is the gap between Hyper-PADA and the T5-NoDA model, which does not perform adaptation: Hyper-PADA outperforms this model by 9.5%. 8.4% and 14.8% in CLCD and CD sentiment classification and CD MNLI, respectively.
These findings support our modeling considera- tions: (1) integrating HNs into OOD generalization modeling (as the HN-based models strongly outper- form the baselines); and (2) integrating DRF sig- nature learning into the modeling framework, both as input to the HN (Hyper-DRY) and Hyper-PADA) and as means of improving example representation (Hyper-PADA). In Appendix C we present addi- tional analysis: (a) Hyper-PADA's performance on seen domains; (b) model performance as function of the training set size; (c) more efficient variations of the Hyper-PADA architecture.
Hyper-DN and Hyper-DRF are also superior to all non-HN models across settings (Hyper-DRF in 10 CLCD sentiment settings, in 7 CD senti- ment settings and in CD MNLI settings, as well as on average all three tasks; Hyper-DN in 7 CD sentiment set- tings, and in 2 CD MNLI settings, as well as on average in all three tasks). It is also interesting to note that the best performing baselines (non- HN models) are different in the three tasks: While T5-MoE (group (a) of domain expert baselines) and T5-DANN (group (b) of domain robustness baselines) are strong in CLCD sentiment classifica- tion, PADA (group (c) of example-based based adaptation basclines) is the strongest baseline for CD MNLI (in CD sentiment classification the average perfor-
To assess the significance of DA approaches in the era of large language models (LLMs), we executed an ablation study, employing the GPT- 3 model (Davinci-003) in a few-shot learning setup, despite the potential data leakage due to the model's likely prior exposure to our test sets (Aiyappa et al., 2023). In this setup, the GPT-3 model was provided comprehensive task descrip- tion, paired with an example for each label from every source domain; for instance, given source domains and m labels, the prompt incorporated k-m fhw-shot examples. Subsequently, we tested
CLCD sentiment settings,

---

F
G
11.
1R
Avg
TS-NoDA
58.2
69.1
IS-MoE-Ind-AAvg
55.6
65.3
57.7
58.1
64.3
60.2
Improvment Over PADA
30
In-Moo-Ind-AAtn
55.6
64.6
TS-MoE-Aug
56.7
66.4
60.0
67.9
65.4
63.3
20
T5-IRM
5L.
61.6
51.7
54.7
61.5
57.3
TS-DANN
72.1
76.9
74.8
76.1
R 0.374
PADA
76.7
79.6
75.3
78.1
75.2
77.0
10
Hyyper-DN
Hyper DRF
75.3
X2.3
73.8
76.3
78.7
77.3
Hlyper PADA
79.04
84.1
78.2
79.8
81.1
80.4
0
Uppershound
30.2
85.8
77.9
$1.5
83.4
0.003
0.004
0.005
0.006
0.007
0.008
Average Standard Deviation of the
Table 3: Cross-domain MNLI results (Macro-F1). The statistical significance of Hyper-PADA vs. the best base- line from each group (with the Bootstrap test, < 0.05) is denoted similarly to Table 2.
Classifier's Weights
Figure 5: Correlation between the diversity of the example-based classifier weights generated by Hyper- PADA, and the improvement of this model over PADA in CLCD sentiment classification. Each point in the graph represents target domain. To estimate the SD, we calculated the SD of each of the weights of the HNs generated for the test examples of this domain, and ported the average. The Spearman Correlation is 0.475. For CD sentiment classification, the corresponding num- bers are 0.539 and 0.175, for Pearson and Spearman correlations, respectively (not shown in the graph).
this 175B parameter model on our CD MNLI and CLCD sentiment analysis tasks. For CD MNLI, GPT-3 yielded an average F1 score of 75.1, a de- cline of 5.3% relative to Hyper-PADA, Conversely, in the sentiment analysis task GPT-3 achieved an average accuracy of 90.1, outperforming Hyper- PADA by 5.6%. Despite any potential impact of test data leakage, the results indicate that LLMs still have some distance cover before their dis- criminative abilities can outstrip fine-tuned models. These findings underscore the continuous need for in-depth DA methods research. See Appendix D.2 for experiment details.
6 Discussion
We presented a Hypernetwork- -based framework for example-based domain adaptation, designed for multi-source adaptation to unseen domains. Our framework provides several novelties: (a) the appli- cation of hypernetworks to unsupervised domain adaptation and any domain adaptation in NLP; (b) the application of hypernetworks example-based manner (which is novel at least NLP, to the best of our knowledge); (c) the generation of example- based classifier weights, based on a learned signa- ture which embeds the input example in the seman- tic space spanned by the source domains; and (d) the integration of all the above with example rep- resentation mechanism that is based on the learned signature. While the concept of DRF signatures stems from Ben-David al. (2022), the aforemen- tioned innovations are unique to our work. Com- prehensive experiments across tasks, 4 languages, 8 domains, and 29 adaptation settings underline our framework's superiority over previous method- ologies and the value of our modeling choices.
Importance of Diversity in Generated Weights To demonstrate the impact of example-based classi- fier parametrization, Figure 5 plots the diversity of the example-based classifier weights as gen- erated by Hyper-PADA vs. the improvement of Hyper-PADA over PADA in the CLCD sentiment classification settings. We choose to compare these models because both of them use the self-generated signature for improved example representation, but only Hyper-PADA uses it for classifier parametriza- tion. To estimate the diversity of the weights gen- erated by the HN in a given test domain, we first measure the standard deviation of each weight gen- erated by the HN across the test examples of that test domain. We then average the SDs of these weights and report the resulting number as the di- versity of the HN-generated weights in the test domain. We repeat this process for each test do- main. The relatively high correlations between the two measures is an encouraging indication, sug- gesting the potential importance of example- based parametrization for improved task performance.
Limitations
Extending beyond sequence classification Al- though our experimental setup broad and ex- tensive, the tasks we considered are limited sentence-level classification tasks. However, there are many other NLP tasks that present challenging

---

out-of-distribution scenarios. Since it is not trivial extend HNs effectively to token-level classifica- tion or text generation, we would like to address such cases in future work. Ultimately, our goal is to shape our methodology to the level that NLP technology becomes available to as many textual domains as possible, with minimum data annota- tion and collection efforts.
the 2006 Conference on Empirical Methods Natu- ral Language Processing, 22-23 July 2006, Sydney, Australia, pages 120-128. ACI..
Tom B. Brown, Benjamin Mann. Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Merbert-Vosss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Bemer. Sam McCandlish. Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. CoRR, abs/20055. 14165.
Utilizing large models Our modeling solution consists of a large pretrained language model. While one could apply the same method using smaller models (available today), it might lead to an unsatisfying performance level compared to the ones reported in this work.
Nitay Calderon, Eyal Ben- BeviDavid, Amir Feder, and Roi Reichart. 2022. Docogen: Domain counterfactual generation for low resource domain adaptation. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), ACL. 2022, Dublin, Ireland, May 22-27, 2022, pages 7727-7746. Association for Computa- tional Linguisties.
References
Rachith Aiyappa, Jisun An, Haewoon Kwak, and Yong- Yeol Ahn. 2023. Can we trust the evaluation on chatgpt? CoRR, abs/2303.12767.
Martín Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. 2019, Invariant risk minimization. CoRR, 0tt//997.02888
Alexis Conneau and Guillaume Lample. 2019. Cross- lingual language model pretraining. In Advances in Neural Information Processing Systems 32: An- nual Conference Neural Information P Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 7057-7067.
Trapit Bansal, Rishikesh Jha, and Andrew McCallum. 2020. Learning to few-shot learn across diverse nat- ural language classification tasks. In Proceedings of the 28th International Conference on Computa- tional Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020, pages 5108-5123, International Committee on Computational Linguis-
Hal Daumé III. 2007. Frustratingly easy domain adap- tation. In ACL 2007, Proceedings of the 45th Annual Meeting of the Association for Computational Lin- guistics, June 23-30, 2007, Prague, Czech Republic. The Association for Computational Linguisties.
tics.
Eyal Ben-David, Nadav Oved, and Roi Reichart. 2022, Pada: Example-based prompt learning for on-the-fly adaptation to unseen domains. Transactions of the Association for Computational Linguistics, 10:414 433.
Lior Deutsch, Erik Nijkamp, and Yu Yang. 2019, A generative model for sampling high-performance and diverse weights for neural networks. CoRR, abs/1905.02889
Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018. The hitchhiker's guide to testing statistical significance in natural language process- ing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018. Melbourne, Australia, July 15-20, 2018, Vol- une I: Long Papers, pages 1383-1392. Association for Computational Linguistics.
Eyal Ben- David, Carmel Rabinovitz, and Roi Reichart. 2020. PERL: pivot-based domain adaptation for pre-trained deep contextualized embedding models. Trans. Assoc. Comput. Linguistics, 8:504-521.
Shai Devid, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. 2010. A theory of learning from different domains. Mach. Learn., 11011)1151-175
Yaroslav Ganin and Victor S. Lempitsky. 2015. Unsu- pervised domain adaptation by backpropagation. In Proceedings of the 32nd International Conference on Machine Learning, 1CML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Con- ference Proceedings, pages 11001 189. JML.
John Blitzer, Mark Dredze, and Fernando Pereira. 2007. Biographies, bollywood, boom- boxes and blenders: Domain adaptation for sentiment classification. In ACL 2007, Proceedings of the 45th Annual Meet- ing of the Association for Computational Linguistics, June 23-30, 2007, Prague, Czech Republic. The As- sociation for Computational Linguisties.
L. Gillick and Stephen J. Cox. 1989. Some statisti- cal issues in the comparison speech recognition algorithms. In IEFF International Conference on Acoustics, Speech, and Signal Processing, ICASSP '89, Glasgow, Scotland, Muy 23-26, 1989, pages 532-
John Blitzer, Ryan T. McDonald, and Fernando Pereira. 2006. Domain adaptation with structural correspon- dence learning. In EMNLP 2006, Proceedings of
535. IEEE

---

Xavier Glorot, Antoine Bordes, and Yoshua Bengio. 2011. Domain adaptation for large-scale sentiment classification: A deep learning approach. In Proceed- ings of the 28th International Conference on Machine Learning, ICML 2011, Bellevue, Washington, USA, June 28- - July? 2, 2011, pages 513-520. Omnipress.
MA, USA, June 7-12, 2015, pages 4840-4848, IEEE Computer Society.
Sylwester Klocek, Lukasz Maziarka, Maciej Wolezyk, Jacek Tabor, Jakub Nowak, and Marek Smieja. 2019. Hypernetwork functional image representation. In Artificial Neural Networks and Machine Learning- ICANN 2019 28th International Conference on Arti- ficial Neural Networks, Munich, Germany, September 17-19, 2019, Proceedings Workshop and Special Sessions, volume 11731 of Lecture Notes in Com- puter Science, pages 196-510. Springer.
Jiang Guo, Darsh Shah, and Regina Barzilay. 2018. Multi-source domain adaptation with mixture of ex- perts. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages
David Ha, Andrew M. Dai, and Quoe V. Lc. 2017. Hypernetworks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Pro- ceedings. OpenReview.net.
David Krueger, Chin-Wei Huang, Riashat Islam, Ryan Turner, Alexandre Lacoste, and Aaron C. Courville. 2017. Bayesian hypernetworks. CoRR, abs/1710.04459.
Xiaochuang Han and Jacob Eisenstein. 2019. Unsu- pervised domain adaptation of contextualized em- beddings for sequence labeling. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, Novem- ber 3-7, 2019, pages 4237-4247. Association for Computational Linguistics.
Virgile Landeiro and Aron Culotta. 2018. Robust text classification under confounding shift. J. Artif. Intell. Res., 63:391-419.
Entony Lekhtman, Yftah Ziser, and Roi Reichart. 2021. Dilbert: Customized pre-training for domain adapta- tion with category shift, with application aspect extraction. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 219-230.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Ges- mundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficier transfer learning for NLP. In Pro- ceedings of the 36th International Conference on Ma- chine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 2790-2799. PMLR.
Zechun Liu, Haoyuan Mu, Xiangyu Zhang, Zichao Guo, Xin Yang, Kwang-Ting Cheng, and Jian Sun. 2019. Metapruning: Meta learning for automatic neural network channel pruning. In 2019 IEEE/CVF In- ternational Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 November 2, 2019, pages 3295 3304.. IEEE.
Rabech Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson. 2021. Parameter- efficient multi-task fine-tunin tuning for transformers via shared hypemetworks. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguisties and the 11th International Joint Confer- ence Natural Language Processing, ACI/IJCNLP 2021, ( (Volume 1: Long Papers), Virtual Event, Au- gust 1-6, 2021, pages (65-576. Association for Com- putational Linguistics.
Weihua Hu, Gang Niu, Issei Sato, and Masashi Sugiyama. 2018. Does distributionally robust su- pervised learning give robust classifiers? In Pro- ceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan. Stockholm, Sweden, July 10-15, 2018, volume of Proceedings of Machine Learning Research, pages 2034-2000 2042 PMLR.
Young-Bum Kim, Karl Stratos, and Dongchan Kim. 2017. Domain attention with an ensemble of experts. In Proceedings of the 55th Annual Meeting of the As- sociation for Computational Linguisties, ACL 2017, Vancouver, Canada, July 30- August 4, Volume 1: Long Papers, pages 643-653. Association for Com- putational Linguistics.
Yishay Mansour, Mehryar Mohri, and Afshin Ros- tamizadeh. 2008. Domain adaptation with multiple sources. In Advances in Neural Information Process- ing Systems 21, Proceedings of the Twenty-Second Annual Conference on Neural Information Process- ing Systems, Vancouver, British Columbia, Canada, December 8-11, 2008, pages 1041-1048, Curran As- sociates, Inc.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track P Proceedings.
Elliot Meyerson and Risto Mikkkulanenn 2019. Mod- ular universal reparameterizatioo: Deep multi-task learning across diverse domains. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Sys- lems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 7901-7912.
Benjamin Klein, Lior Wolf, and Yehuda Afek. 2015. A dynamic convolutional layer for short rangeweather prediction. In IEEE Conference on Computer Vi- sion and Pattern Recognition, CVPR 2015, Boston,

---

Eliya Nachmani and Lior Wolf. 2019. Hyper-gras network decoders for block codes. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Sys- lems 2019, NewrlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 2326-2334 2336.
Guy Rotman and Roi Reichart. 2019. Deep contex- tualized self-training for low resource dependency parsing. Transactions of the A Association for Compu- tational Linguistics. 7:695-713.
Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. 2020. Distributionally robust neu- ral networks. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Yonatan Oren, Shiori Sagawa, Tatsunori B. Hashimoto, and Percy Liang. 2019. Distributionally robust lan- guage modeling. In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNL.P 2019, Hong Kong. China, November 3-7, 2019, pages 4226 4236. Association for Computational Linguis- tics.
Joan Serrà, Santiago Pascual, and Carlos Segura. 2019. Blow: a single-scale hyperconditioned flow for non- parallel raw-audio voice conversion. In Advances in Neural Information Processing Systems 32: An- nual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 6790-6800
Nick Pawlowski, Martin Rajchl, and Ben Glocker. 2017. Implicit weight uncertainty in neural networks. CoRR, 1.002997.
Aviv Shamsian, Aviv Navon, Ethan Fetaya, and Gal Chechik. 2021. Personalized federated learning us- ing hypernetworks. In Proceedings of the 38th In- ternational Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 148999502... PMLR.
Emmanouil Antonios Platanios, Mrinmaya Sachan, Gra- ham Neubig, and Tom M. Mitchell. 2018 Contextual parameter generation for universal neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Procexs- ing, Brussels, Belgium, October 31- November 4, 2018, pages 425-435, Association for Computational Linguisties.
Falong Shen, Shuicheng Yan. and Gang Zeng 2018. Neural style transfer via meta networks. In P Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8061-8069.
Peter Prettenhofer and Benno Stein. 2010. Cross-
language text classification using structural corre- spondence learning. In ACL 2010, Proceedings of the 48th Annual Meeting of the Association for Com- putational Linguistics, July 11-16, 2010, Uppsala, Sweden, pages 1118-1127. The Association for Com- puter Linguistics.
Joseph Suarez. 2017. Language modeling with recur- rent highway hypernetworks. In Advances in Neural Information Processing Systems 30: Annual Confer- ence on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 3267-3276.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. J. Mach. Learn. Res.,
Ivan Titov. 2011. Domain adaptation by constraining inter-domain variability of latent feature representa- tion. In The 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference, 19-24 June, 2011, Portland. Oregon, USA, pages 62-71. The Association for Computer Linguistics.
Alan Ramponi and Barbara Plank. 2020. Neural un- supervised domain adaptation in NLP- A survey. In Proceedings of the 28th International Confer- ence on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020, pages 5838-6855. International Committee on Com- putational Linguistics.
Kenya Ukai, Takashi Matsubara, and Kuniaki Uehara. 2018. Hypernetorr-- based implicit posterior estima- tion and model averaging of CNN. In Proceedings of The 10th Asian Conference on Machine Learning, ACMI 2018, Beijing, China, November 14-16, 2018 88, volume 95 of Proceedings of Machine Learning Re- search, pages 176-191. PMLR.
Roi Reichart and Ari Rappoport. 2007. Self-training for enhancement and domain adaptation of statistical parsers trained on small datasets. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 616-623.
Ahmet Üstün, Arianna Bisazza, Gosse Bouma, and Gert- jan van Noord. 2020. Udapter: Language adaptation for truly universal dependency parsing. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 2302-2315. Associa- tion for Computational Linguistics.
Gernot Riegler, Samuel Schulter, Matthias Rüther, and Horst Bischof. 2015. Conditioned regression models for non-blind single image super-resolution. In 2015 IEEE International Conference Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pages 522-530. IEEE Computer Society.
Ahmet Üstün, Arianna Bisazza, Gosse Bouma, Gertjan van Noord, and Sebastian Ruder. 2022. Hyper-x:

---

unified hypernetwork for multi-task multilingual transfer. arXiv preprint arXiv:2205. 12148.
Sicheng Zhao, Guangzhi Wang, Shanghang Zhang, Yang Gu, Yaxian Li. Zhichao Song, Pengfei Xu, Runbo Hu, Hua Chai, and Kurt Keutzer. 2020. Multi- source distilling domain adaptation. In The Thirry- Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirly-Second Innovative Appli- cations of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 12975- 12983. AAAI Press.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention all you need. In Advances in neural information pro- cessing systems, pages 5998-6008.
Ashwin K. Vijayakumar. Michael Cogswell, Ram- prasaath R. Selvaraju, Qing Sun, Stefan Lee, David J. Crandall, and Dhruv Batra. 2016. Diverse beam search: Decoding diverse solutions from neural se- quence models. CoRR, abs/1610.02224.
Yftah Ziser and Roi Reichart. 2017. Neural structural correspondence learning for domain adaptation. In Proceedings of the 21st Conference on Computa- tional Natural Language Learning (CoNLL 2017), Vancouver, Canada, August 3-4, 2017, pages 400- 410. Association for Computational Linguistics.
Johannes von Oswald, Christian Henning, João Sacra- mento, and Benjamin F. Grewe. 2020. Continual learning with hypemetworks. In 8th Interattional Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia. April 26-30, 2020. OpenRe- view.net.
Yftah Ziser and Roi Reichart. 2018. Deep pivot-based modeling for cross-language cross-domain transfer with minimal guidance. In Proceedings of the 2018 Conference on Empirical Methods in Natural Lan- guage Processing, Brussels, Belgium, October 31 November 4, 2018, pages 238-249. Association for Computational Linguistics.
Yoav Wald, Amir Feder, Daniel Greenfeld, and Uri Shalit. 2021. On calibration and out-of-domain gen- cralization. In Thirty-Fifth Conference on Neural Information Processing Systems.
Adina Williams, Nikita Nangia, and Samuel R. Bow- man. 2018. broad-coverage challenge corpus for sentence understanding through inference. In Pro- ceedings of the 2018 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL- HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume (Long Papers), pages 1112-1122. Association for Computational Linguistics.
Yftah Ziser and Roi Reichart. 2019. Task refinement learning for improved accuracy and stability of un- supervised domain adaptation. In proceedings the 57th annual meeting of the Association for Computa- tional Linguistics, pages 5895-5906.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistae, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of-the--e -he natural language processing. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Dustin Wright and Isabelle Augenstein. 2020. Trans- former based multi-source domain adaptation. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 7974.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mt5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pages 483-498. Association for Computational Linguistics.

---

A Additional Background
Sentence. This documentary pourly produced. has terrible sound quality and stereotypical "life affirming" stories. There was nothing in here support Wal- their business practices or their philosophy. Domain, DVD.
A.1 Domain Related Features (DRFs)
In order to perform example-based domain adapta- tion, the first stage of the Hyper-DRF and Hyper- PADA models maps each input example into se- quence of Domain Related Features (DRFs). Se- lecting the DRF sets of the source domains is hence crucial for these models, as they should allow the models to map input examples to the semantic space of the source domains. Since a key goal of example-based adaptation is to account for soft domain boundaries, it is important that the DRF set of each source domain should reflect both the unique semantic aspects of this domain and the aspects shares with other source domains.
Label. Negative. DRF Signature. music: history, rock, sound, story
Table 4: An example of Hyper-DRE and Hyper-PADA application to a sentiment classification example. The source domains are Books, and Music. Generated DRF features from the Books and Music domains are in blue and green, respectively.
Hyper-DRF and Hyper-PADA we have to construct a DRF signature for each training example. Our goal in this process is to match each training exam- ple with those DRFs in its domain's 's DRF set that are most representative of its semantics. We do this in automatic manner.
To achieve these goals, we follow the definitions, selection, and annotation processes Ben-Davin et al. (2022). For completeness, we briefly describe these ideas here.
Let wi; W., be the tokens of sentence from the domain Sj. Each DRF ry E Rj is assigned with the following score:
DRF Set Construction Let S be the set of all source domains, and Sj S the domain for which we construct the DRF set. We perform the follow- ing selection process, considering all the training data from the participating source domains. First, we define the domain label of a sentence to be 1 if the sentence is from Sj and O otherwise. We then look for the top / words with the highest mutual in- formation (MI) with the 0/1 labels. Then, since MI could indicate association with each of the labels (related to the domain (1) or not (0)), and we are in- terested only in words associated with the domain, we select only words that meet the criterion:
score(rj, (Wi, E [x) min {s(r),w;) wi) ==
s(rj, ws) = (ry)) ((w))11
where (xx) is the embedding of x in the pre-trained embedding layer of an off-the-shelf BERT model. Then, let Ti,..... Th be the k DRFs with the lowest scores and D the domain name. We define the DRF signature of x to be the following string: "D Ti TT...,TT:"
To summarize, we utilize this annotation only during training, as a training signal for the DRF signature generator (in stage of both Hyper-DRF and Hyper-PADA).
Cs)sy(w) S p, (w) > 0 (ss(w)
Tables 1 (main paper) and 4 provide MNLI and sentiment classification examples and their DRF signatures, as generated by dlpeerPAADA and Hyper-DRL DRF in specific adaptation setup.
Where Csvs, (w) is the count of the word w in all of the source domains except Sj. Cs, (w) is the word count in Sj and p is domain-specifiiity parameter: The smaller it is, the stronger is the association. The DRF set of Sj is denoted with Ry.
B Dataset Sizes
Table 5 presents the number of training, develop- ment and test examples from each domain. Notice that since we consider multiple training domains in each of our experiments, the number of training and development examples in our experiments are an aggregation of the numbers shown in the table. For example, in the CLCD sentiment analysis task, when we test on the English DVD domain, we use 3000 training examples, 600 development exam- ples and 2000 test examples. In each experiment,
Annotating DRF- Signatures for Training In order to train the DRF signature generator of
"We also implemented an alternative approach which ex- tracts DRF sets based on a TF-IDF criterion. Yet, we noticed that the extracted DRFs are very similar to the ones extracted by the method of Ben-David et al. (2022), which we use for the main results of this paper, and so are the downstream task performances. For instance, in the MNLI task, the average performance differences between implementations with the two DRF selection methods, for Hyper-DRe DRF and Hyper-PAD. PADA are 0.1% and 0.6%, respoctively.

---

Sentiment Analysis (En, De, Ft. Jp)
Sentiment CLCD
Sentiment CD
MILI
Domain
Training (sre)
Dev (sre)
Test (trg)
Books (B)
500
100
2000
IS-NoDA
78.7
$2.0
65.2
I5-MoE-Ind-Arg
80.4
DVD (D)
300
100
2000
I5-MoE-Ind-Attu
84.7
84,0
59.9
Music (M)
300
2000
TS-MoF-Avg
83.6
80.0
61.9
MNLI (En)
TS-IRM
77.1
81.8
57.1
Domain
Training (src)
Dev (src)
Test (trg)
TS-DANN
81.3
79,0
72.2
Fiction (F)
2500
200
1,973
PADA
78.6
83.0
77.1
Hyper-DN
86.7
85.7
77.1
Government (G)
2500
200
1, 9.945
HyperrDRRF
77.7
Slate (SL)
2500
200
1, 1,955
Hyper-PADA
87.5
85,5
80.6
TT)))
2500
200
1,966
Travel (TR)
2500
200
1,976
Table 6: Seen domains results. HN-based methods are superior.
Table 5: The number examples in each domain (and language) of our two tasks. We denote the examples used when a domain included as source domain (src), and when it the target domain (trg). For sentiment we present the number of examples in a single language, while there are four different languages English (En), Deutsch (De), French (Fr), and Japanese (Jp), each with the same number of examples per domain.
subsets corresponding to subsets larger than 30% (with 8 subset points). Each point in the presented graphs presents the average performance across all settings. For instance, the point corresponding 10% in CLCD sentiment analysis presents the erage performance across all CLCD settings (12 overall). Accordingly, each of the 12 settings uses 10% of the training examples of the corresponding
the source domains development sets are used in order to select the hyper- paraa--prraeterss of the models.
source domains (we sample a subset of the 10% from each source domain).
Ablation Analysis
For sentiment classification, Figure 6a presents clear and stable trend across all subsets: Hyper- PADA is superior all three baselines. The perfor- mance gap between the methods is more significant in low-resource scenarios (smaller training subsets). Furthermore, while Hyper-PADA's advantage de- creases as the labeled training size grows, it still performs better than its baselines across all training set sizes.
Training Size Effect Our experiments focus scenarios that are both low-resource and domain adaptation, as the combination of the two yields very challenging, yet realistic, generalization setup (Landeiro and Culotta, 2018; Calderon et al., 2022). Yet, it is also essential to assess the impact of our modeling approach across training sets of various sizes, including cases where labeled data is abun- dant. Hence, next turn evaluate Hlper-PADA it compares to other baselines, T5-NoDA, T5- DANN and T5-IRM, across multiple subsets of the training data available for our tasks (sentiment anal- ysis and MNLI). We experiment with the following subset sizes: 10%-100% (in 10% steps) for the CLCD setting (sentiment analysis); and 1%-5% (in 1% steps) and 10%-100% (in 10% steps) for the CD setting of MNLI. For each experiment, sample subset of the corresponding percentage from the training examples of each of the source domains and use the same test and validation sets across all experiments.
For MNLI, when considering up to 20% of the data (more than 60K training examples), Hyper- PADA significantly outperforms all three baseline, as demonstrated in Figure 6b. For larger sub- sets (more than 30%, Figure 6c), Hyper-PADA, T5-DANN and T5-NoDA demonstrate compatible performance, while T5-IRM reaches significantly lower results. We note that for subsets of 30% of the MNLI data, the models train on more than 22.5K examples from each source domain (for a total of 90K training examples), which seems to be enough to overcome the OOD effect. For com- parison, the 100% subsets of the CLCD sentiment analysis dataset contain 12K examples.
Figure 6 summarizes our results. Figure 6a presents sentiment classification results for the CLCD transfer, including subsets ranging from 10% to 100% (for total of 10 subset points). Fig- ure 6b presents results for MNLI in the CD transfer, with subsets ranging from 1% to 20% (with sub- set points) and Figure 6c focuses on the MNLI
Eval Evaluating Performance on Seen Domains In this paper, we put strong emphasis on the perfor- mance of algorithm on unseen target domains. Our main reasoning is that compared to the lim- ited number of known source domains, there is potentially an unlimited number of unknown tar-

---

80
Berrage Actunity
Average Mecrr
20
Aversge Macm
83.5
63.0
PADA
Ryper
TS-NoDA
60
TB-NoDA
62.5-
TS-NIDDA
TS-DANN
T5-DAMN
62.0
TS-DANN
T5-RM
T5:RM
20
40
6n
100
10
26
40
100
100
Porcentage Data
Percentage of Data
Percentage of Data
(a) Sentiment Analysis- 10%
(b) MNLI to
(c) MNLI 30%
Figure 6: The performances of Hyper-PADA and T5-NoDA on training subsets of different size. The red vertical dashed lines present the training subset size in our main experimental setup.
#15
Model
G
Avg
Dual vs. Solo TS Model Performance In the ar- chitectures of both Hyper-PADA and Hyper-DRE, employed distinet T5 models. One served as signature generator, while the other, trained from scratch, functioned as an encoder for the dis- criminative component (refer to Figure main paper). As an alternative, one could consider using single TS model to perform both roles. In this approach, the training regimen alternates be- tween signature generation and classification tasks (mediated by the hypemetwork). Each training example stands 20% probability of being used for generation. As evidenced in Table 7, the dual T5 model setup consistently delivers superior per- formance across all model sizes compared to the single-model approach.
Size
2
Small
69.3
75.6
65.6
69.1
71.7
70.3
Small
67.1
71.9
61.6
67.6
67.0
2
Base
79.0
84.1
78.2
79.8
81.1
Large
85.6
88.1
79.8
85.9
86.8
85.2
the
Table 7: The results Hyper-PADA using one or two T5 models. The models version is the one described in the main paper.
#HN Layers
AParams
G
TL
TR
Arg
2.4M
79.2
83.7
76.5
80.2
79.1
79.7
30M
78.2
79.8
81.1
3.5M
77.8
83.4
77.7
79.5
80.9
79.9
Table 8: MNLI F1 results of Hyper-PAPae with differ- ent number of fully connected layers (1,2,3). For each configuration, we present the number of corresponding HN parameters.
HN Architectures Variants We subsequently sessed the best configuration for the HN, explicitly focusing on the number layers within the HN. The main paper discussed the results of an HN with a single layer responsible for generating classifier weights. In this analysis, we experimented with varying layer counts: 1, 2, and 3, incorporating a ReLU activation after each layer. The findings presented in Table 8. While the single-layer setup isn't 't optimal, adding more layers doesn't 't offer a substantial improvement.
get domains, which the algorithm may encounter in future tests. Still, it is essential to verify that our algorithms do not sacrifice their source domain performance in order to achieve -ntoo--distribution generalization. Hence, We next measure the perfor- mance the source domains in each experiment by calculating the Fl score (MNLI) or accuracy (sentiment classification) across all development examples from the source domains. In each ex- periment, we calculate the relevant metric on each source domain's 's validation set. Then, we average the results of each domain across all runs.
D Implementation Details
D.1 URLs of Code and Data
Our Code Repository our official code repository will be published upon acceptance. In addition, attach to this submission zip folder that contains our anonymized code source.
Table 6 reports our results, demonstrating the
superiority of our models seen domains. The HN models superior in all the setups, with Hyper-PADA outperforming all models for senti- ment CLCD and MNLI setups and is the second best model in the sentiment CD setup, where is slightly outperformed by Hyper-DN.
HuggingFace (Wolf et al., 2020) code and pretrained weights for the T5 model and tok-

---

enizer: https://hugging.
MNLI
Sentiment
Model
Train
Total
Train
Total
MNLI dataset The natural language in- ference data experimented with within this paper. https://cing http//cisssyu.......m eduu/////bb/wmn//// multinli/
No-DA
110
110
277
277
MOE-Ind
439
439
1662
1662
MOE-Avg
548
548
1939
1939
IRM
110
110
277
277
DANN
110
110
277
277
PADA
333
442
859
1027
Websis-CLS-I dataset The multilingual multi-domain dataset which is experimented with within this paper. org/reccrdd3251167tt YdQiIWBBWS
Hyper-DN 112
221
280
447
Hyper-DRF
335
444
862
1030
Hyper-PADA 335
444
862
1030
Table 9: The number in millions of parameters in each model. MOE-IND represents both MOE-IND-Avg and MOE-IND-Atta since the difference is negligible.
D.2 Text classification with GPT3
This section claborates on the methodology em- ployed for text classification with GPT3 within our DA experiments, and provides insight into the implementation process of GPT3 prompting. Our focus is conducting DA from various source domains to unknown target domains. Given this scenario, zero-shot GPT3 setup naturally aligns with our paradigm, albeit assuming that the origi- nal GPT3 training phase devoid of data contam- ination. However, this approach is generally less effective than the few-shot GPT3 alternative.
(Xue et al., 2021) for sentiment classification. For contextual representation of the HN input (domain name or "UNK* in Hyper-DN, DRF signature in Hyper-DRI and Hyper-PADA) we use the BERT- base-uncased and the mBERT-based els, for MNLI and sentiment classification, respec- tively.
mod-
We choose p = 1.5 for the DRF set construction process. In the DRF signature annotation process, take the ko 5 most associated DRFs for each input example. When generating the signature (in Hyper-DRF and Hyper-PADA) we employ the Di- verse Beam Search algorithm (Vijayakumar et al., 2016) with the T5 decoder, using the following pa- rameters: 5 sequences, with a beam size of 5, a 5 beams group and a diversity penalty of 0.1.
To incorporate the few-shot approach of GPT-3 while adhering to experimental protocol, limit the model employ only instances from the source domain within each experiment. Our prelim- inary studies showed enhanced performance when task instructions and specifics were added to assist the model in comprehending the challenge it faces.
Thus, we augment our prompt with task-related knowledge, instructions, and examples from multi- ple domains. These examples feature one example for each label from every source domain. It's worth noting that while all test examples within a domain- shift experiment employ an identical prompt, we modify the prompt for each distinct domain-shift of the same task, given it incorporates examples from varied domains.
The HN consists of two linear layers of the same input and output dimensions (1 x 768), each of which followed by ReLU activation layer. The output the second layer is fed into two parallel linear layers, one to predict the weights of the linear classifier (a 2 x 768 matrix), and one to predict its bias (a vector). For task classification, we feed the linear classifier (CLS) with the average of the encoder token representations.
For transparency and reproducibility, in Table 10 we provide the exact prompt designed for the MNLI experiment. As part of this process, we set the temperature parameter to zero. Similarly, our CLCD sentiment analysis prompt designed following these same guidelines.
Generative models are trained for 3 epochs and discriminative models for 5 epochs. We use the Cross Entropy loss for all models, optimized with the ADAM optimizer (Kingma and Ba, 2015), batch size of 16, and learning rate of 5 10-6. We limit the number of input tokens 128.
D.3 Hyperparameter Different Choices
D.3.1 Computing Infrastructure and Runtime
For all the pre-trained models we use the Hugging- face Transformers library (Wolf et al., 2020). For the T5 model we use the T5-base model (Raffel et al., 2020) for MNLI, and the MT5-base model
All experiments were performed on a single Nvidia Quadro RTX 6000 GPU, with 4608 cores, 24 GB GPU memory, 12 CPU cores and 125 GB RAM. For a single CLCD sentiment analysis experiment

---

Table 10: GPT3 prompt for CD MNLI.
with Hyper-DN, we measured runtime of min- utes, which corresponds single cell in Table 2 (in the Hyper-DN row). Respectively, for sin- gle CD MNLI experiment, we measured a runtime of 12 minutes for Hyper-DN, corresponding to a single cell in Table 3. For Hyper- PADA and Hyper- DRF, we measured runtime of 20 minutes for single CLCD sentiment analysis experiment, and 45 minutes for single MNLI experiment (corre- sponding to single cell Table and a single cell Table respectively). table 9 we present the number of parameters in each the models and baselines used this paper. While our model has large number of parameter due to the use of T5 encoder-decoder and separate T5 encoder, other methods use significantly larger (TS-MOE versions) comparable number (PADA) while reaching lower results.
## Task:
Classify the following sentence pair, the 'premise' and "hypothesis', into three categories: Entailment,
Contradiction, and Neutral. The premise begins with 'First: ', the hypothesis begins with 'Second: ' and they are seperated with 

## Guidelines:
1. ##Entailmentxx. If the premise is true, the hypothesis is definitely true.
#*Contradiction=:: If the premise
is true, the hypothesis is definitely false.
**Neutral**: The premise doesn't definitively confirm refute the hypothesis.
#* Tips:
Understand the context of both sentences before deciding. Look for key words or phrases that indicate the relationship.
Don't make assumptions based on outside knowledge. The premise alone should dictate your decision.
If a word has different meanings in the premise and hypothesis, can change the relationship.
Cont iootinyy learn from feedback and ask for clarification when needed.
н* Example 1:
isrc_domain_@. ___t_all]
Answer: Entailment
#* Example 2: dora_ddmmin___cc contrrdiii))
Answer: Contradiction
## Example 3: dec___dmain___neutttee Answer: Neutral
#* Example 11: Answer: Contradiction
## Example (src_domain___neutrre Answer: Neutral
Now your turn нæ Example 13:
(example)
Answer:


Reviews:
[{"review_id": "3AQzdDR581", "reviewer": "Reviewer_cySR", "summary": "", "questions": "1. Authors state \"hypernetworks in example-based manner, which is novel at least in NLP, to the best of our knowledge\". So, is it", "limitations": "", "rating": 3, "confidence": 2, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**General response to all reviewers:**\\\nWe thank the reviewers for their insightful comments.\nAs acknowledged by most reviewers, we address an important and challenging task of adaptation to unknown domains under a low resource regime. This task has been explored in the literature, and the best-performing approaches are PADA and MoE. Our proposed methods achieve strong results in extensive experimentation, outperforming PADA and MoE by impressive gaps. \nWe would like to list again our main algorithmic contributions in short:\n1. The use of Hypernetworks (HNs) for adaptation to unseen domains (i.e. domains that are unknown at training time).\n2. The application of HNs in an example-based manner. Using our mechanism, the HNs’ output is tailored to each example, which allows a more flexible model and improves performance.\n3. The integration of the prompting mechanism into a hyper-network-based framework, in a way that yields SOTA adaptation performance.\n\n**Response to reviewer cySR:**\\\nWe appreciate the positive feedback regarding the quality of writing, the novelty of the idea, and the significance of the problem addressed. We have carefully considered your concerns and provide the following comments:\n\n1. Regarding the naive method: \n\nRecapping our method: Hyper-PADA uses a T5 encoder-decoder to generate a unique (example specific) semantic signature. Then, this signature is fed as a prefix of the example into the T5 encoder, and as an input to the HN which generates the classifier weights for this specific example. Finally, the classifier predicts the task label conditioned on the encoding vectors of the ‘prompted’ input.\n   \nThe development of our method required nuanced intuition and a deep understanding of the problem, informed by extensive familiarity with prior works. Moreover, the uniqueness of our approach in this context underlines its innovation.  \n\nFinally, and perhaps most importantly, our results are very strong, across tasks, domains and languages, which is indicative of the quality of our model. As our ablation analysis shows, many alternatives to our model perform much worse.\n\n2. Alternative methods like disentanglement:  \nDisentanglement approaches may serve as an efficient way to learn domain-specific and domain-invariant properties of the data. Following this comment, we adjusted the implementation of [1] to fit our setup. Specifically, we learn domain-generic and domain-specific spaces and use both to predict the task label. The domain generic space is learned with a domain adversarial loss (instead of NSP, as in [1], to fit the domain scenario) and the domain-specific space is learned in a similar fashion as in [1], through domain classifier (instead of task classifier). We tested this approach on our MNLI snd Sentiment CLCD (comparing to our models):\n\nMNLI (CD)\\\n| |  F    |  G    |  S    |  TL  |  TR | AVG |\\\n+-------------------+-----+-----+-----+-----+-----+-----+\\\n| Disentangle |69.5 |77.1 |67.8  |71.9  |72.8 |71.8 |\\\n| Hyper-PADA |79.0 |84.1  |78.2  |79.8  |81.1  |80.4 |\\\n+-------------------+-----+-----+-----+-----+-----+-----+\n\n\nSentiment (CLCD)\\\n|                |Deutsch             |English             |French              |Japanese           | AVG |\\\n|                |  B  |  D  |  M  |  B  |  D  |  M  |  B  |  D  |  M  |  B  |  D  |  M  |      |\\\n+----------------+------+------+------+------+------+------+------+------+------+------+------+------+\\\n| Disentangle |82.7 |80.0 |82.6 |87.3 |82.7 |83.4 |80.2 |53.8 |80.2 |68.2 |82.9 |58.2 |76.8 |\\\n| Hyper-PADA |85.7 |81.8 |85.0 |86.0 |84.4 |85.1 |86.8 |85.9 |81.8 |83.9 |83.9 |83.8 |84.5 |\\\n+----------------+------+------+------+------+------+------+------+------+------+------+------+------+\n\n\nAs can be seen, our models are superior to this approach. \n\n\n3. Visualization & interpretability:  \nWe kindly refer the reviewer to the experiments presented in Appendix C. Specifically, in lines 1262-1283 (Figure 6), we reveal the significance of diversity in the weights generated by the HN, highlighting the mechanism that  allows Hyper-PADA to outperform PADA. \nFollowing this comment, we will take two steps:\\\n(A) We will move into the extra page of main paper (allowed in the camera ready version) the following analyses currently in the appendix.\n- The importance of diversity in generated weights (Fig 6) \n- The effect of training size on performance (Fig 5) \n- Performance metrics for seen domains (Table 6) \n\n(B) To shed light on the success of our method for different HN architectures, we add an experiment considering 1/2/3 layers HNs. Our results:\n\nMNLI (CD)\\\n|                 | #HN Layers | #params|  F  |  G  |  S  |  TL |  TR | AVG |\\\n+------------------+-----------+--------+-----+-----+-----+-----+-----+-----+\\\n| Hyper-PADA      |     1     | 2.4M   |79.2 |83.7 |76.5 |80.2 |79.1 |79.7 |\\\n| Hyper-PADA      |     2     | 3.0M   |79.0 |84.1 |78.2 |79.8 |81.1 |80.4 |\\\n| Hyper-PADA      |     3     | 3.5M   |77.8 |83.4 |77.7 |79.5 |80.9 |79.9 |\n\n\n** Questions:\nYes, we are the first to present an example-based hypernetwork in NLP.\n\n[1] “Continual Learning for Text Classification with Information Disentanglement Based Regularization”, Huang et al., 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well-written.\n\n2. The idea seems to be interesting. \n\n3. The paper studies an important problem. ", "reasons_to_reject": "1. The proposed method is a little naive, which combines T5 encoder-decoder with a hypernetwork. \n\n2. There could be some different methods to learn shared and domain-specific aspects of the input such as disentanglement. Why the proposed is better?\n\n3. More visualization and interpretability and  should be provided to validate the effectiveness of the hypernetwork. ", "questions_to_authors": "1. Authors state \"hypernetworks in example-based manner, which is novel at least in NLP, to the best of our knowledge\". So, is it", "paper_topic_and_main_contributions": "This paper addresses the issue of multi-source adaptation for unfamiliar domains: We leverage labeled data from multiple source domains to generalize to unknown target domains at training. Authors employ example-based Hypernetwork adaptation: a T5 encoder-decoder initially generates a unique signature from an input example, embedding it within the source domains' semantic space.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "W7f3gOVJ9H", "reviewer": "Reviewer_StMx", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**General response to all reviewers:**\\\nWe thank the reviewers for their insightful comments.\nAs acknowledged by most reviewers, we address an important and challenging task of adaptation to unknown domains under a low resource regime. This task has been explored in the literature, and the best-performing approaches are PADA and MoE. Our proposed methods achieve strong results in extensive experimentation, outperforming PADA and MoE by impressive gaps. \nWe would like to list again our main algorithmic contributions in short:\n1. The use of Hypernetworks (HNs) for adaptation to unseen domains (i.e. domains that are unknown at training time).\n2. The application of HNs in an example-based manner. Using our mechanism, the HNs’ output is tailored to each example, which allows a more flexible model and improves performance.\n3. The integration of the prompting mechanism into a hyper-network-based framework, in a way that yields SOTA adaptation performance.\n\n\n**Response to reviewer StMx:**\\\nWe appreciate your recognition of our work's novelty and its thorough empirical evaluation. Below, we address the concerns raised. We note that the first two concerns raised by the reviewer were also discussed in our paper, under the Limitations section.\n\n1. Limited to Classification Tasks:\n   \nWe stress that our setup choice, adapting to unfamiliar domains and languages, is already advanced. As our experiments demonstrate, many well-established domain adaptation methods do not perform well in our scenarios. Diving into other tasks, like token classification, would introduce more complexities. For instance, token-based classification would force discerning the nuances of applying hypernetworks at the token level and the heightened challenge of adapting to unseen domains for token-based tasks. While these challenges are undeniably pivotal, addressing them would demand dedicated research that might shift the focus of our current paper. \n\nNotice also that our work addresses two tasks, a large number of domains and both cross-domain and cross-language adaptation.  This is already a much more extensive experimental setup than in most domain adaptation work. In the PADA paper, for example, which has an unusally extensive experimental setup, there is a token classifcation task, but no cross-language generalization is demonstrated; Indeed in our paper we show that PADA does not perform well in the cross-language cross-domain adaptation setup, and our HN-based models perform much better.\n\n Nonetheless, we acknowledge the significance of this comment and have earmarked it for future exploration.\n\n2. Requires Large Pretrained Models:\n\nTo put things in perspective, in the age of hundreds of billions parameter models (one of them, GPT3.5, is serving as a baseline to our models), our largest model employs two-scale fewer parameters. Furthermore, MoE, one of the previous SOTA models for our setup, requires more parameters than ours. \n\nFollowing this comment, we experimented with the following models, testing the effect of making our models more efficient:\n- Hyper-PADA with a single T5-large (instead of two T5-base).\n- Hyper-PADA with a single T5-base.\n- Hyper-DRF with a single T5-small.\n- Hyper-PADA with two T5-small architectures.\n\nOverall, large models improve end-to-end results, and using two models is superior to using one. However, our model is superior to all baselines even with a single T5-base. For convenience, we report the results achieved with Hyper-PADA variants and compare them to those documented in the paper:\n\nMNLI (CD)\\\n|                 | #T5   | model size |  F  |  G  |  S  |  TL |  TR | AVG |\\\n+------------------+-------+------------+-----+-----+-----+-----+-----+-----+\\\n| Hyper-PADA      |   2   |   small    |67.1 |71.9 |61.6 |67.6 |67.0 |67.0 |\\\n| Hyper-PADA      |   1   |   Small    |69.3 |75.6 |65.6 |69.1 |71.7 |70.3 |\\\n+------------------+-------+------------+-----+-----+-----+-----+-----+-----+\\\n| Hyper-PADA      |   2   |   Base     |79.0 |84.1 |78.2 |79.8 |81.1 |80.4 |\\\n| Hyper-PADA      |   1   |   Base     |78.7 |84.1 |76.8 |78.1 |80.7 |79.7 |\\\n+------------------+-------+------------+-----+-----+-----+-----+-----+-----+\\\n| Hyper-PADA      |   2   |   Large    |85.4 |88.1 |83.7 |85.8 |88.1 |86.2 |\\\n| Hyper-PADA      |   1   |   Large    |85.6 |88.1 |79.8 |85.9 |86.8 |85.2 |\n\n3. Incremental Contribution: \nPrevious works in the intersection of HNs and DA in NLP assume limited labeled data in the target domain. Using this data, with much-labeled data in other domains, the HN learns to generate the set of parameters for the target domain. Notice that this setup is arguably not a domain adaptation challenge in the context of emerging few-shot learning models. Hence, this contribution is very limited compared to ours.\n\nOur contribution over prior HN approaches is three-fold:\n-  Our method is tailored for scenarios without prior target domain knowledge. This presents unique challenges, especially in determining the appropriate weights the hypernetwork should generate. Notice that the number of papers that addressed our domain adaptation scenario is limited and our results are substantially stronger.\n- We integrate into our model a 'signature' generation mechanism for each example (borrowed from the PADA paper), and this signature is fed both to the representation componenet and to the HN. This enables a novel application of an HN-based model in an example-based manner  - something that has not been done in the past in the field of NLP. The resulting strong performance are much better than those of the original PADA algorithm.\n- We are the first to present an example-based hypernetwork approach (that is, even regardless of our contribution to out-of-distribution generalization), particularly in NLP.\n\n We will make sure to highlight these contributions in the final version, given the opportunity.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Novel application of hypernetworks for domain adaptation in NLP: The paper presents an innovative way to leverage hypernetworks for parameter sharing in domain adaptation scenarios. This opens up new possibilities for using hypernetworks in NLP.\n2. Thorough empirical evaluation: The models are evaluated extensively across multiple domain adaptation setups over two tasks involving both cross-domain and cross-lingual transfer. This provides convincing evidence of the benefits of the proposed approach.", "reasons_to_reject": "1. Limited to classification tasks: The experiments focus only on text classification tasks. It is unclear if the approach would transfer well to other NLP tasks like sequence labeling, generation, etc. Expanding the evaluation to other tasks could strengthen the paper.\n2. Requires large pretrained models: The approach relies on large pretrained encoder-decoder models like T5, which have significant computational requirements. Especially for Hyper-DRF  and Hyper-PADA, they require an additional T5 model. Exploring ways to make the approach more efficient could broaden its applicability.\n3. Incremental contribution: The core idea of using hypernetworks for domain adaptation is not entirely new, though the specific application to unseen target domains in NLP is novel. The incremental contribution over prior hypernetwork adaptation approaches should be clarified.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a new approach using hypernetworks for domain adaptation in natural language processing. The key contributions are:\n\n1. Proposes three models (Hyper-DN, Hyper-DRF, Hyper-PADA) using hypernetworks for domain adaptation, allowing parameter sharing at the domain and example levels. This is a novel application of hypernetworks in NLP.\n2. Introduces an example-based adaptation method where the model generates a \"signature\" for each example to map it to the semantic space of the source domains. This signature is used as input to the hypernetwork and to modify the input representation.\n3. Evaluates the models extensively on sentiment analysis and natural language inference across multiple domains and languages. The results show the hypernetwork models outperform previous domain adaptation methods.\n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "PKphkq1EfS", "reviewer": "Reviewer_4JUK", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**General response to all reviewers:**\\\nWe thank the reviewers for their insightful comments.\nAs acknowledged by most reviewers, we address an important and challenging task of adaptation to unknown domains under a low resource regime. This task has been explored in the literature, and the best-performing approaches are PADA and MoE. Our proposed methods achieve strong results in extensive experimentation, outperforming PADA and MoE by impressive gaps. \nWe would like to list again our main algorithmic contributions in short:\n1. The use of Hypernetworks (HNs) for adaptation to unseen domains (i.e. domains that are unknown at training time).\n2. The application of HNs in an example-based manner. Using our mechanism, the HNs’ output is tailored to each example, which allows a more flexible model and improves performance.\n3. The integration of the prompting mechanism into a hyper-network-based framework, in a way that yields SOTA adaptation performance.\n\n\n**Response to reviewer 4JUK:**\\\nWe appreciate your recognition of our task's importance, the improvements our models present, and our work's thorough empirical evaluation. We believe that addressing your concerns will sharpen our message and improve our paper. Below, we address the concerns raised:\n\n1. Vague arguments\n- Lines 89-101 - Thank you. Indeed we meant that the statement is correct in the current context. We will clarify the scope, and novelty, and refer to [1].\n- Lines 283-294 - We think our argument was not understood. Following this comment, we will replace the vague explanation in line  283 with: “We hypothesize that parameter sharing in the domain level may lead to suboptimal performance. Following [2], we present the Hyper-DRF model.\n\n2. Choice of the dataset.\nIndeed, the famous Amazon dataset for domain adaptation is an example of a very large dataset with a diverse set of domains. Yet, this dataset is exceptional as it is tied to a major application of one oof the world's largest companies and the labels are generated by Amazon's users. Hence, we still think that our argument is correct for the much more common scenario: when addressing a new task, one typically has a limited annotation budget (caused by budget/time/data-availability). In such cases, they practically have to choose between a large number of examples from a single domain and a smaller number of examples from several domains. We will clarify that this is not always the case, and cite projects like the Amazon sentiment analysis dataset.\n\n\n[1] Zhao, Sicheng, et al. \"Multi-source distilling domain adaptation.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34. No. 07. 2020.\n[2] Ben-David, Eyal, et al. \"PADA: Example-based Prompt Learning for on-the-fly Adaptation to Unseen Domains.\"", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The task is practical. Typographically, the paper is well-written and easy to understand. The models are straightforward, and the improvements are convincing. The experiments are also thorough.", "reasons_to_reject": "The methods are very straightforward—in a good way—not much can be said about them, but some of the arguments in the paper are weak, vague, and incorrect. In Lines 89-101 the authors claim that “there is no DA method that focuses on domain invariant and domain specific features simultaneously.” \nThere has been more more than 20 years of research on domain adaptation, of course there has been studies on that!! See [1]. The reality is that because the task that the authors are working on is very new, they can claim that in that specific task this idea has not been used—but yet no other ideas has been used either, the task is new anyway.\n\nIn Lines 283-294, the authors use the argument that “because weight sharing may not be sufficient and because domain boundaries are not always well defined,” then they propose a model to learn from the target examples.\nI don’t understand this argument. If you are arguing that a model or solution is not working well, you need to reveal its weakness. Show its flaws. Not just use vague words. Domain adaptation is a theoretical field. Indeed there have been theories that prove there exists a function consisting of the weighted average of the source distributions that has arbitrary low target error [2]. That function may be achieved by weight sharing.\nIf you have an idea that works, and you do not know why it works, please just say you don’t know why it works. Avoid giving vague arguments.\n\nPlease avoid forcing reviewer to read appendix, see Line 320.\n\nLines 408-417, I am not convinced by the authors argument about the choice of dataset. Please see the newer version of the famous amazon dataset for domain adaptation [3].\n\n\n\n[1] Zhao, Sicheng, et al. \"Multi-source distilling domain adaptation.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34. No. 07. 2020.\n\n[2] Mansour, Yishay, Mehryar Mohri, and Afshin Rostamizadeh. \"Domain adaptation with multiple sources.\" Advances in neural information processing systems 21 (2008).\n\n[3] https://www.cs.jhu.edu/~mdredze/datasets/sentiment/", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors propose solutions for a multiple source domain adaptation scenario where the target domain is not accessible during the training. Their solutions are based on hyper-networks. In the first case the hyper-network learns a representation for each domain and generates the weights of the classifier. In the second case, the hyper-network learns a representation for each target example using a T5 model and generates the classifier weights. And in the third case the hyper-network learns the representation and a prompt.\n\nThey test their models in two tasks and 4 languages and show that it outperforms some naive baselines—the task is fairly new, so not many baselines are available.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "[1] Zhao, Sicheng, et al. \"Multi-source distilling domain adaptation.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34. No. 07. 2020.\n\n[2] Mansour, Yishay, Mehryar Mohri, and Afshin Rostamizadeh. \"Domain adaptation with multiple sources.\" Advances in neural information processing systems 21 (2008).\n\n[3] https://www.cs.jhu.edu/~mdredze/datasets/sentiment/"}}, {"review_id": "wO71eVvaHY", "reviewer": "Reviewer_suhr", "summary": "", "questions": "- What intrinsic mechanism in hypernetworks makes them useful?\n- Does the architecture of hypernetworks affect the performance a lot?\n- You mention that PADA is the current state-of-the-art model. But it performs not well in CLCD setting (Table 2). Any possible reasons?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**General response to all reviewers:**\\\nWe thank the reviewers for their insightful comments.\nAs acknowledged by most reviewers, we address an important and challenging task of adaptation to unknown domains under a low resource regime. This task has been explored in the literature, and the best-performing approaches are PADA and MoE. Our proposed methods achieve strong results in extensive experimentation, outperforming PADA and MoE by impressive gaps. \nWe would like to list again our main algorithmic contributions in short:\n1. The use of Hypernetworks (HNs) for adaptation to unseen domains (i.e. domains that are unknown at training time).\n2. The application of HNs in an example-based manner. Using our mechanism, the HNs’ output is tailored to each example, which allows a more flexible model and improves performance.\n3. The integration of the prompting mechanism into a hyper-network-based framework, in a way that yields SOTA adaptation performance.\n\n**Response to reviewer suhr:**\n\n1. The motivation for Hypernetworks:\nThe effectiveness of hypernetworks in domain adaptation can be attributed to their ability to transcend discrete domain boundaries and embrace a continuous perspective. Unlike conventional methods that often focus solely on one domain, hypernetworks enable a more holistic approach, allowing each sample to draw upon the collective knowledge of all pertinent domains (an approach like MoE that does consider multiple domains, imposes strict domain boundaries).  In our implementation, the knowledge share is at the example level,  where each example is represented through a mixture of domain-related semantic components. \n\n2. The motivation for Prompts:\nAs demonstrated in extensive NLP research, prompts help put things in context for the model. In our case, the prompt aims to put the input example (from an unfamiliar domain) in a semantic context that the model recognizes (I.e. as noted above, the prompt represents the example as a mixture of domain-related semantic components).\n\nIn practice, the prompts are used to condition the discriminative model on the specific domain attributes presented in them. For instance, when the prompt is made of the domain name (like in the Hyper-DN model), the classifier is conditioned on the domain and learns different P(Y|X;D) distributions for each domain D. When the prompt contains DFRs (like in Hyper-DRF and Hyper-PADA), each example is represented through  an example-specific mixture of source domain properties. This forces the classifier to learn a more complex conditional distribution and enables the classifier to generalize to examples from unseen domains. \n\nThe PADA paper explores this issue extensively, providing evidence and analysis of the importance of the prompt. For sanity, we checked for similar behaviors in our methods (e.g., what happens when the prompt consists of random tokens), and observed similar trends to those reported in PADA. However, we do not think that these analyses should be presented in our paper, as we would like to save valuable space for novel material. Given the opportunity we will address this in the final version.\n\n\n3. The effect of the hypernetwork architecture:\nFollowing this comment, we experimented with 3 different architectural configurations and measured the variation of scores. \nThe architecture of the HN in the paper consisted of two linear layers with an additional activation on top of each. To estimate the effect of the architecture on the downstream results, we also test a 1-layer-based HN and a 3-layer-based HN. We report the results of this experiment in the following table, focusing on the MNLI task.\n\nMNLI (CD)\\\n| | #HN Layers | #params| F | G | S | TL | TR | AVG |\\\n+------------------+-----------+--------+-----+-----+-----+-----+-----+-----+\\\n| Hyper-PADA | 1 | 2.4M |79.2 |83.7 |76.5 |80.2 |79.1 |79.7 |\\\n| Hyper-PADA | 2 | 3.0M |79.0 |84.1 |78.2 |79.8 |81.1 |80.4 |\\\n| Hyper-PADA | 3 | 3.5M |77.8 |83.4 |77.7 |79.5 |80.9 |79.9 |\\\n+------------------+-----------+--------+-----+-----+-----+-----+-----+-----+\n\n\n\n As demonstrated in the above table, the effect of the HN architecture exists, yet is not prominent, at least with the experimented changes. The robustness to hyper-parameter configurations is hence another upside of our method. \n\n\n4. PADA’s weak performance in CLCD:\n\nOur method combines the advantages of both prompting and hypernetworks, while PADA utilizes only prompting. In the CLCD setup, which was not part of the PADA paper, it's evident that hypernetworks are more effective than prompting, based on the following observations:\n1. Our hypernetwork (HN) models significantly outperform all models that do not employ an HN.\n2. There's only a minor performance difference between our HN-based  models with varying prompting strategies.\n3. As noted by the reviewer, PADA performs poorly on this task. We believe the diversity of languages disrupts PADA's prompting mechanism, thus affecting its efficiency.\n\n\nNotice that our CLCD setup - adaptation to unseen domains and languages - have not been explored in the past in the NLP literature (to the best of our knowledge). Hence, uncovering the reduced performance of PADA in this setup is a contribution of our paper. Yet, as PADA is so strong in the cross-domain setup, it was a straightforward decision to consider it for CLCD as well. We will make this clear in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The improvements are promising.\n- Considering hypernetworks is interesting.", "reasons_to_reject": "- The motivation is not clear enough for me. Although the experimental results look good, the authors should answer the question that why hypernetworks help generalization. What intrinsic mechanism in hypernetworks makes them useful?\n- It's unclear for me that why adding prompt help the generalization. I know it's proposed by the PADA paper, but I suggest the authors provide more insights or experiments to study more about it.", "questions_to_authors": "- What intrinsic mechanism in hypernetworks makes them useful?\n- Does the architecture of hypernetworks affect the performance a lot?\n- You mention that PADA is the current state-of-the-art model. But it performs not well in CLCD setting (Table 2). Any possible reasons?", "paper_topic_and_main_contributions": "This paper focuses on domain adaptation tasks and aims to make model generalize better for unseen domains. The key idea is using hypernetworks to generate the classifier weights. They also provide two variations: with example-based domain related features (DRF) and with example-based DRF + prompt. Experimental results show the effectiveness of the proposed methods.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None.", "Typos_Grammar_Style_and_Presentation_Improvements": "None."}, "extra_text": {}}]

Author rebuttal:

Meta-review: All the reviewers agree that this method paper is well-written, works on a practical problem, and presents a convincing improvement for unseen domain adaptation given the extensive experimental setup (across domains, tasks, and languages). One common weakness of the paper are the motivation for the usage of hypernetworks, which manifests in different ways such as "Why not consider disentanglement" and "Why hypernetworks help generalization?"  In my opinion, I think that the authors have done a good job in the rebuttals in motivating the use of hypernetworks, explaining the advantages of their method, and showing the results comparing their proposed method and the disentanglement method mentioned by reviewer cySR. The performance gap between their proposed method and the existing methods such as PADA and MoE is quite impressive.

While the reviewers raise other issues on the simplicity of the approach and the lack of other types of tasks and datasets, in my opinion, this paper has already demonstrated a promising approach to tackle a non-trivial task that they set out to explore and carried out extensive experiments including ablation study to verify the strength of their proposed approach. While it is undesirable that many findings are in the Appendix, such as the ablation and interpretability experiments, this paper is information-heavy and yet the authors have done a good job in articulating the settings and their important results. However, certain arguments in the paper could have been strengthened as suggested by reviewer 4JUK. The motivation for the proposed approach should also be detailed further in the revised version of the paper.
Decision: Accept-Findings