Universal Domain Adaptation for Robust Handling of Distributional Shifts in NLP
Authors: Hyuhng Joon Kim
Hyunsoo Cho
Sang-Woo Lee
Junyeob Kim
Choonghyun Park
Sang-goo Lee
Kang Min Yoo
Taeuk Kim
Venue: EMNLP 2023 Findings (2023)
Keywords: Domain Adaptation
Universal Domain Adaptation
Out-of-Distribution Detection

Abstract: When deploying machine learning systems to the wild, it is highly desirable for them to effectively leverage prior knowledge to the unfamiliar domain while also firing alarms to anomalous inputs. In order to address these requirements, Universal Domain Adaptation (UniDA) has emerged as a novel research area in computer vision, focusing on achieving both adaptation ability and robustness (i.e., the ability to detect out-of-distribution samples). While UniDA has led significant progress in computer vision, its application on language input still needs to be explored despite its feasibility. In this paper, we propose a comprehensive benchmark for natural language that offers thorough viewpoints of the model’s generalizability and robustness. Our benchmark encompasses multiple datasets with varying difficulty levels and characteristics, including temporal shifts and diverse domains. On top of our testbed, we validate existing UniDA methods from computer vision and state-of-the-art domain adaptation techniques from NLP literature, yielding valuable findings: We observe that UniDA methods originally designed for image input can be effectively transferred to the natural language domain while also underscoring the effect of adaptation difficulty in determining the model’s performance.

Universal Domain Adaptation for Robust Handling of Distributional Shifts in NLP
Hyuhng Joon Kim', Hyunsoo Cho Sang-Woo Lee Junyeob Kim', Choonghyun Park', Sang-goo Lee Kang Min Yoo! 1,2,3, Taeuk Kim
'Souul National University, "NAVER Cloud, "NAVER Al Lab,
*KAIST, IIteelliSys, Hanyang University
(heyjoonkim, johyunsoo johyunsoo,jjny111555 pch3300 wwwgeggeeeeeooooo snu. snu.ac. kr (sang. woo lee, venee kimtaeuk@hanyangggkk kc.
Abstract
Training Phase
When deploying machine learning systems to the wild, it highly desirable for them to effec- tively leverage prior knowledge to the unfamil- iar domain while also firing alarms anoma- lous inputs. In order address these require- ments, Universal Domain Adaptation (UniDA) has emerged as a novel research area com- puter vision, focusing achieving both adap- tation ability and robustness (i.e., the ability detect out-of-distribution samples). While UniDA has led significant progress in computer vision, its application on language input still needs be explored despite its feasibility. In this paper, we propose comprehensive bench- mark for natural language that offers thorough viewpoints of the model's generalizability and robustness. Our benchmark encompasses mul- tiple datasets with varying difficulty levels and characteristics, including temporal shifts and diverse domains. On top of our testbed, we val- idate existing UniDA methods from computer vision and wate-of-theear domain adaptation techniques from NLP literature, yielding valu- able findings: We observe that UniDA meth- ods originally designed for image input can be effectively transferred the natural language domain while also underscoring the effect of adaptation difficulty in determining the model's performance.
Please open the window.
Real World
Hey, mind opening the window for me? Thanksl
Sure. N open the window.
Whip sandwich real quick?
Sorry, cannot handle your request.
The model trained with formal language (source domain) will likely face spoken language (target domain) in the real world. The model is expected to properly handle such transferable input despite the dis- tributional shift. (middle) At the same time, the model should discern unprocessable inputs (bottom) from the target domain.
Figure 1:
in a significant drop in performance (Ribeiro et al., 2020; Miller et al., 2020: Hendrycks et al., 2020). The aforementioned phenomenon represents long- standing challenge within the machine learning community, wherein even recent cutting-edge lan- guage models (OpenAI, 2023; Touvron al., 2023; Chowdhery et al., 2022; Brown al., 2020) do not serve as an exception to this predicament (Wang et al., 2023).
1
Introduction
Deep learning models demonstrate satisfactory per- formance when tested on data from the training distribution. However, real-world inputs encounter novel data ceaselessly that deviate from the trained distribution, commonly known as distributional shift. When confronted with such inputs, machine learning models frequently struggle to differentiate them from regular input. Consequently, they face challenges in adapting their previously acquired knowledge to the new data distribution, resulting
In response to these challenges, existing lit- crature proposes two distinct approaches. The first approach, known Domain Adaptation (DA) (Blitzer al., 2006; Ganin al., 2016b; Karouzos et al., 2021; Wu and Shi, 2022), endeavors to estab- lish alignment between new set of data from an unknown distribution and the model's prior knowl-
CCo-corresponding authors.

---

edge distribution. The objective is enhance the model's generalization capability and reduce per- formance drop springing from the distributional shift. In parallel, a distinct line of work, referred to as out-of-distribution (OOD) detection (Aggarwal, 2017: Hendrycks and Gimpel, 2017; Hendrycks et al., 2019; Cho et al., 2021), focuses on discern- ing imputs originating from dissimilar distributions. They opt circumvent potential risks or disrup- tions arising from shifted inputs, thereby enriching system robustness and resilience.
of the problem, i.e., OOD detection and DA. Our empirical results show that UniDA meth- ods are fully transferable the NLP domain and can robustly respond to various degrees of shift. Moreover, we find out that the adaptation difficulty notably affects the performance of the methods. In certain circumstances, DA methods display compa- rable even better performance. We release our dataset, encouraging future research UniDA in NLP foster the development of more resilient and domain-specific strategies.
While both approaches offer unique advantages addressing specific distributional shifts, integrating their merits could substantially enhance robustness. In pursuit of this objective, a novel field called Uni- versal Domain Adaptation (UniDA) (You et al., 2019) has emerged, aiming to harness the syner- gies of both OOD detection and DA when con- fronted with distributional shifts. UniDA leverages the best of the two worlds and offers comprehen- sive perspectives that integrate the merits of these two research areas. The essence of UniDA lies in measuring the uncertainty of the data from the shifted distribution precisely. Then, we can en- hance the model's 's transferability by distinguishing portions of low-uncertainty inputs that can be ad- equately handled with the current model's knowl- edge. Simultaneously, we enrich the robustness of the model OOD inputs by discerning the re- maining samples that cannot be processed normally. However, distinguishing between these inputs and properly processing them becomes increasingly challenging without explicit supervision.
Universal Domain Adaptation
2.1 Problem Formulation
Distributional shift refers to a situation where the joint distribution P estimated from the training dataset fails to adequately represent the wide range of diverse and complex test inputs. More formally, distributional shift arises when the test input Best originates from a distant distribution Q, which is not effectively encompassed by the current trained distribution
One the most prevalent research areas address- ing this distributional shift includes OOD detection and DA. OOD detection aims to strictly detect all inputs from Q to enhance the model's 's reliability. Although distribution Q demonstrates a discernibly different distribution from the distribution P. the trained model can still transfer subset of instances from Q, overcoming the inherent discrepancy be- tween P and Q. This particular capability serves as a fundamental motivation underlying the pursuit of DA. UniDA endeavours to integrate the merits of both fields, thereby enhancing both the gener- alizability and reliability of the model. Specifi- cally, let us divide the target distribution Q into dis- joint subsets H. which share the same label space with source distribution P and its complement I QQ = H U I). The objective of UniDA to enrich the robustness of the model by flcxibly transferring existing knowledge to transferable samples from H while firing alarms to unknown samples from I.
Despite the versatility of UniDA, this topic has yet to be explored in the Natural Language Process- ing (NLP) literature. As cornerstone in enhancing reliability against distributional shifts in NLP, we introduce testhed for evaluating the model's ro- bustness in a holistic view. First, we construct vari- ous adaptation scenarios in NLP, utilizing an array of thoughtfully selected datasets. To discern the degree to which our proposed datasets incorporate the various degree of challenges in UniDA, we de- fine two novel metrics: Performance Drop Rate (PDR) and Distinction Difficulty Score (DDS). Using these metries, We verify that our testbed captures a broad spectrum of distributional shifts. Finally, based on the suggested setting, we sys- tematically compare several UniDA methods in- herently designed for the task, against heuristic combinations of previous approaches for the parts
2.2 Challenges in UniDA
UniDA models should be capable of accurately capturing the underlying reasons behind the shift, thereby enabling the discrimination between frans- ferable samples and unknown samples. Among the diverse categories of causes, the domain gap
"The dataset is available at https:://ittub..... com/ reyyjonniimuniivrsse www.mmiinnnaaaaaiioooforr.o.ooa

---

and the category gap (You et al.. 2019) emerge as pivotal factors, each exerting substantial impact on the overall complexity of the UniDA problem. While these concepts have previously been defined in a rather vague manner, we deduced the necessity for more explicit definition. Thus, we set forth to redefine the concepts of domain and category gap more explicitly.
3.1 Quantifying Different Shifts
As the extent of both domain and category gaps significantly influences the overall adaptation com- plexity, it is essential to quantify these gaps when designing the dataset for evaluation. Unfortunately, existing literature has not devised a clear-cut and quantitative measure for assessing domain and cat- egory gaps. Therefore, we endeavoured define measures that can aptly approximate the two types of gaps.
Domain gap refers to the performance drop when a model trained on P fails to correctly process transferable inputs due to the fundamental discrep- ancy between P and H, i.e., a domain shift. A dataset with a higher domain gap amplifies the problem's s difficulty as the trained model becomes more susceptible to misaligning transferable sam- ples.
Performance Drop Rate (PDR) measures the degree of domain gap by assessing the absolute difficulty of the dataset itself and the performance drop caused by the shift from P to H. Specifically, fine-tune
on the source train set and evaluate its test set accuracy acc, from the same distribution. Leveraging the same model trained on the source domain, we then measure the accuracy of the target test set accc. We measure the performance degradation caused by the distri- butional shift by measuring acc, acc; Since the significance of the performance drop may vary de- pending the source performance, we normalize the result with the source performance and mea- sure the proportion of the performance degradation. A more significant drop rate indicates greater decline in performance, considering the source do- main performance. Formally, PDR for a source domain s and target domain can be measured as follows:
category shift, characterized by the disparity in the class sets considered by P and I, causes a cate- gory gap. Category gap represents the performance drop that arises for inputs from I, which cannot be processed properly due to differing class sets be- tween P and I, which are erroneously handled. A larger category gap makes distinguishing unknown samples from transferable samples harder, thereby worsening the robustness of the model.
From the perspective of addressing the domain gap and category gap, the main goal of UniDA is to minimize both gaps simultaneously. This aims to ensure transferable samples properly align with the source domain for adequate processing, while handling unknown samples as unprocessable excep- tions.
PDR,, = 100 X OCC, acet
(1)
aCC₂
Distinction Difficulty Score (DDS) is measured
3 Testbed Design
to estimate the difficulty of distinguishing between H and I, which, in other words, measures the dif- ficulty of handling the category shift. We utilized the same model trained on the source domain and extracted the [CLS] representations of the source inputs. We estimated the source distribution, as- suming the extracted representations follow the multivariate normal distribution. We then extracted [CLS] representations of target distribution inputs from the same model and measured the Maha- lanobis distance between the source distribution. Using the distance, we measured the Area Under the ROC Curve (AUC) metric for discerning I and H. AUC values closer indicate the ease of discerning unknown inputs from the transferable inputs. Since we focus on quantifying the difficulty in distinguishing the two, we subtract the AUC from to derive our final measure of interest. For
The primary objective of our research is con- struct a comprehensive benchmark dataset that ef- fectively captures the viewpoint of UniDA. To ac- complish our objective, we attempt to create a di- verse dataset that encompasses a range of difficulty levels and characteristics, such as domains, senti- ment, or temporal change. These variations are the fundamental elements that can significantly influ- ence the overall performance.
Specifically, we initially select datasets from multiple practical domains and approximate the adaptation difficulty by quantifying different shifts with our newly proposed metrics. In the following subsections, we provide an in-depth explanation of our dataset along with the analysis of our bench- marks.

---

the source domain s, the target domain L, and AUC , DDS can be measured as:
Huffpost (2013)
Huffpost (2014)
Huffpost (2015)
Huffposst (2016)
Dutaset
PDR
7.87
20.69
20.47
25.00
DDS = 100 x (1 a
(2)
DDS
19.67
27.79
25.71
31.19
Huffpost (2017)
Dataset
CLINC-150
MASSIVE
Amazon
3.2 Implementation of Different Shifts
To construct a representative testbed for UniDA, is essential to illustrate domain and category shifts. To exhibit domain shift, we delineated domains from various perspectives. This involves explicit factors such temporal sentiment and implicit definitions based the composition of the class set. Detailed formation of domains for each dataset is stipulated in Section 3.3.
PDR
33.62
11.59
36,14
15.45
DDS
31.74
13:45
23.92
36,04
Table I: PDR and DDS values of each datasets. The largest value of PDR and DDS highlighted in bold.
rating of 4 5 are categorized as positive. We exclude 3-star reviews, considering them neutral. MASSIVE (FitzGerald et al., 2022) is hi- erarchical dataset for intent classification. The dataset consists of 18 first-level and 60 second- level classes. Each domain is defined set of classes, including private classes exclusive to spe- cific domain and common classes shared across domains. We divided the common first-level class into two parts based on second-level classes to sim- ulate domain discrepancy. The half of the divided common class represents the source domain while the other half represents the target domain. We as- sume that the second-level classes within the same first-level class share common feature and thus can be adapted.
To establish category shifts, the source and the target domain must have a set of common classes C and a set of their own private classes, Cg and Ör, re- spectively. We followed previous works (You et al., 2019; Fu et al., 2020) by sorting the class name in alphabetical order and selecting the first |C] classes common, the subsequent [C,] as source private, and the rest as target private classes. The class splits for each dataset are stated as |C1/1C,I/ICa) in the main experiments.
3.3 Dataset Details
We focused on text classification tasks for our ex- periments. Four datasets were selected from mul- tiple widely used classification domains in NLP, such as topic classification, sentiment analysis, and intent classification. We reformulated the datasets that our testbed could cover diverse adaptation scenarios.
CLINC-150 (Larson et al., 2019) is widely used for intent classification in OOD detection. The dataset consists of 150 second-level classes over 10 first level-classes and single out-of-scope class. The domain is defined in the same way as MAS- SIVE.
Huffpost News Topic Classification (Huffpost) (Misra, 2022) contains Huffpost news headlines spanning from 2012 to 2022. The task is to clas- sify news categories given the headlines. Using the temporal information additionally provided, we split the dataset year-wise from 2012 to 2017, treat- ing each year as distinct domain. We selected the year 2012 as the source domain, with the subse- quent years assigned as the target domains, creating 5 different levels of temporal shifts.
3.4 Dataset Analysis
In this section, we intend to validate whether our testbed successfully demonstrates diverse adapta- tion difficulties, aligning with our original motiva- tion. We assess adaptation difficulty from domain and category gap perspectives, each approximated by PDR and DDS, respectively.
The results of PDR and DDS are reported in Ta- ble 1. The result shows diverse PDR values ranging from 7 to 36 points, indicating various degrees of domain gap across the datasets. MASSIVE mea- sured the most considerable domain gap, while Huffpost (2013) demonstrated the most negligi- ble domain gap among the datasets. Additionally, our testbed covers a wide range of category gaps, indicated by the broad spectrum of DDS values. Specifically, Amazon exhibits a significantly high
Multilingual A Amazon Reviews Corpus (Ama- zon) (Keung et al., 2020) includes product reviews that are commonly used to predict star ratings based the review, and additional product information is provided for each review. We have revised the task to predict the product information given the reviews and utilized the star ratings to define senti- ment domains. Reviews with a star rating of or are grouped as negative sentiment, and those with

---

CLINC-150 150
H2f1p
and AdSPT (Wu and Shi, 2022), under the assump- tion that all the inputs from the target domain are transferable without considering unknown classes. Two previous state-of-the-an art UniDA methods were selected, OVANet (Saito and Saenko, 2021) and UniOT (Chang et al., 2022), which are fully op- timized to handle UniDA scenarios in the vision domain. We also conducted experiments with ad- ditional baseline methods such DANN (Ganin et al., 2016a), UAN (You et al., 2019), and CMU (Fu et al., 2020). However, the performance was subpar compared to the selected methods, exhibit- ing a similar tendency. Hence, we report the ad- ditional results in Appendix A. For the backbone of all the methods, we utilized bert- pert-bass-uncased (Devlin et al., 2019) and used the [CLS] represen- tation the input feature. Implementation details are stipulated in Appendix B.
Huffpost(2013)
Huffpost(201))
Huffposti201
MASSIVE
Huffposti2015)
Amazon
30
DDS
25
35
PDR
Figure 2: The visualization of adaptation difficulty for each dataset in terms of domain gap (PDR) and category gap (DDS). We categorized the dataset into distinct groups based on the adaptation complexity. Best viewed in color.
4.2 Thresholding Method
Since CDA methods are not designed handle unknown inputs, additional techniques are required to discern them. A straightforward yet intuitive approach to detecting unknown inputs is applying a threshold for the output of the scoring function.
DDS value, representing extremely challenging scenario of differentiating the unknown samples from the transferable samples.
We consolidate the two indicators to measure the coverage of different adaptation complexity of proposed testbed. We visualized the datasets considering the PDR and DDS; metrics on the re- spective axes. Figure is the visualization of the adaptation complexity for each dataset factored by PDR and DDS. We grouped the datasets into four distinct clusters based on the plotted distri- bution. Datasets closer to the lower-left corner, CLINC-150 and Huffpost (2013), are easy adap- tation scenarios with minor domain and category gaps. Datasets plotted in the center, Huffpost (2014, 2015, 2016), presents moderate difficulty. Amazon suffers from a category gap significantly, while Huffpost (2017) and MASSIVE demonstrate a no- table domain gap, yielding high adaptation com- plexity. The results validate that our testbed em- bodies diverse range of adaptation difficulties as intended.
The scoring function reflects the appropriateness of the input based on the extracted representation. If the output of the scoring function falls below the threshold, the instance is classified as unknown. We sequentially apply thresholding after the adaptation process." Formally, for an input æ, categorical pre- diction ÿ. threshold value w, and scoring function fscores the final prediction is made as:
2(x) argmax(()) if Sacore (x) unknown, otherwise.
(3)
We utilize Maximum Softmax Probability (MSP) as the scoring function" (Hendrycks and Gimpel, 2017). Following the OOD detection literature, the value at the point of 95% from the sorted score values was selected the threshold.
'In the case applying thresholding first, all the puts from the target domain would classified OOD. all the target inputs were classified OOD, the criteria for dis- cerning trunsferable
4 Experimental Setting
4.1 Compared Methods
wnknown inputs become inherently unelear. Therefore, have only considered scenarios where thresholding is applied after adaptation.
We compare several domain adaptation methods on our proposed testbed. We selected two previ- ous state-of -fate-of-the-an thet-arttt closed-set Domain Adaptation (CDA) methods, UDALM (Karouzos et al., 2021)
'In addition to MSP. we have applied various scoring func- tions such as cosine similarity and Mahalanobis distance, but using MSP achieved the best performance. Results for other thresholding methods have heen included in the Appendix A.

---

AdSPT (CDA)
UnioT (UniDA)
Huffpost (2012 + 2013) (3/4/4)
Method
accc
0000
H-score
UDALM
52.74 #2,92
58,55
#6,03
55.15 #2.37
AdSPT
55.05 4:2.11
80.66
2:2.9
65.38 4:100
H-score
OVANet
65.11 4:0.6
24.91
16.75
35.64
16.70
40
UniOT
53.76 +1.10
65.86
13357
59.14 +1.14
CLINC-150 (4/3 /3)
20
Method
aceC,
H-score
UDALM
74.92 +6.11
69.91
+:3.98
71.28 +:4.62
CLINC-150
2013
2014
2015
2016
2017
MASSIVE
Amazon
AdSPT
3,96 +:217
98.69
+.036
7.55
4-3.95
OVANet
83.49 4.062
31.24
1.770
45.45
1:1.8
UniOT
64.14 #9.14
77.36
#3.94
69.88 #6.11
Dataset
Table 2:
Experimental results on CLINC-150 and Huff- post (2013), which is relatively easy adaptation sce- nario. The best method with the highest H-score is in bold, and the second- bccond-best method is underlined.
Figure 3: H-score results of AdSPT and UniOT on all the datasets. The preferred method varies depending the adaptation complexity.
4.3 Evaluation Protocol
and UniDA approach: AdSPT representing CDA and UniOT representing UniDA. Despite an outlier caused by unstable thresholding in CLINC-150, the overall trend demonstrates that AdSPT manifests comparable performance in less complex scenar- ios, while UniOT exhibits superior performance towards challenging scenarios. These trends align with the findings of other methods that are not de- picted in the figure.
The goal of UniDA is to properly process the trans- ferable inputs and detect the unknown inputs simul- taneously, consequently making both the transfer- able and the unknown accuracies crucial metrics. We applied H-score (Fu et al., 2020) as the pri- mary evaluation metric to integrate both evaluation metries. H-score the harmonic mcan between the accuracy of common class accc and unknown class accc,, where acce is the accuracy over the common class set C and acco, is the accuracy pre- dicting the unknown class. The model with high H-score is considered robust in the UniDA setting, indicating its proficiency in both adaptation (high acec) and OOD detection (high accc.). Formally, the H-score can be defined as
5.2 Detailed Results
Table 2 demonstrates the results of relatively easy adaptation scenarios. CDA methods demonstrate performance that is par with, even superior to, UniDA methods. The results appear counter- intuitive, as CDA methods are designed with- out considering unknown samples. Specifically, UDALM outperforms all the UniDA methods CLINC-150 and performs comparable or even bet- ter in Huffpost (2013). AdSPT exhibits the best performance in Huffpost (2013). However, AdSPT suffers significant performance drop in CLINC- 150, as we speculate this result due to the inherent instability of the thresholding method. The mis- guided threshold classifies the majority of the in- puts as unknown, which leads to a very high acce, but significantly reduces the accg. This inconsis- tency also leads to high variance of acce, for all the CDA methods.
Hscore = 2 acce accc,
(4)
accc accc,
Although the H-score serves an effective eval- uation criterion for UniDA, we also report accc and accc, to provide comprehensive assessment. We report the averaged results and standard devia- tions over four runs for all experiments.
5 Experimental Results
5.1 Overview
We conduct evaluations based the clusters de- fined in Section 3.4 and analyze how the results vary depending on the adaptation complexity. Fig- ure 3 presents an overview of the H-score results for the berf-perfonning method from each CDA
In the case of moderate shifts, no particular method decisively stands out, as presented by Ta- ble 3. In all cases, AdSPT and UniOT present

---

Huffpost (2012 - 2014) (3 4)/4)
Amazon (11 10/ 10)
Method
acec
0000,
H-score
Method
accc
a00c,
H-score
UDALM
29.90 +1.89
50.00
#17.49
34. 14 #5.07
UDALM
47.69 #1.47
15.10
#2.30
22.85
#2.65
AdSPT
29.64 1:4.1
80.03
66.09
42.93 43.41
AdSPT
30.26 #3.52
30.93
+26.14
27.12
#11.03
OVANet
45.85
+:2.2
33.96
16778
36.02 43.95
OVANet
44,50
#0.84
42.49
#3.95 5
43.38
#1.8 1.88
UniOT
33.49 1.4.99
71.44 19119
45.23 43.60
UniOT
38.60 #1.08
70.98 77.76
49,90 1:
Huffpost (2012 + 2015) (3 / 4/
Table 4:
Experimental results on Amazon, which has significant influence of category gap. The best method with the highest H-score are in bold, and the second best method underlined.
Method
accc
o00c,
H-score
UDALM
25.15
+:260
61.79
88888
35.56
+33115
AdSPT
29.08 +:5.99
73.04
+:2.71
40.78
+3.42
OVANet
43.91
4:2.09
35.70
1:4.1
39.21
42.43
MASSIVE (8 5/)
UniOT
27.16
+3.40
75.50
=2.68
39.82
+3.39
Huffpost (2012
2016) (3 / 4 4)
Method
accc
H-score
Method
accc
accC,
H-score
UDALM
38.33 +4.88
41.29
+15.14
39.28
19.09
AdSPT
36.78 +2.65
59.87
+13.7
45.32 26.20
UDALM
29.14 +1.59
60.87 66.87
39.28 +1.44
AdSPT
28.84 +3.37
82.52
+4.22
42.55
+3.27
OVANet
59.04 #0.69
39.28
#2.14
47.12 +1.77
UniOT
44.01 +:0.20
79.19
+:3.2
56.52 +33.49
OVANet
44.30 +2.89
33.49
#4.79
38.03
#3.79
UniOT
33.09 4:362
69.27
+3.83
44.60 +:2.85
Huffpost (2012
2017)
(3 / 4/4
Method
acep
0000,
H-score
Experimental results on Huffpost (2014, 2015, 2016), which has a moderate complexity for adaptation. The best method with the highest H-score is in bold, and the second-best method is underlined.
Table 3:
UDALM
32.85 +1.05
43.08
7.
37.08 #2.82
AdSPT
35.74 #3.33
53.04
+16.02
41.36 #5.04
OVANet
48.37 1:412
35.03
#3.60
40.46
4:2.32
UniOT
31.57 +1.92
75.85
+9.63
44.36 :: 1:227
the best performance with a marginal difference, making it inconclusive to determine a superior ap- proach. Despite the relatively subpar performance, UDALM and OVANet also exhibit similar results. Still, it is notable that CDA methods, which are not inherently optimized for UniDA settings, show comparable results.
Table 5: Experimental results on MASSIVE and Huff- post (2017), which demonstrates high domain gap. The best method with the highest H-score is in bold, and the second- best method is underlined.
but the gap marginal. Even though CDA methods pose comparable performance, UniDA methods demonstrate better overall performance.
The result of Amazon, in which the category gap is most prominent, is reported in Table UniDA methods exhibit substantially superior performance to CDA methods. In particular, while the difference in accc is marginal, there exists a substantial gap of up to 55 points in accc, As the category gap intensifies, we observe the decline in the perfor- mance of CDA methods, which are fundamentally limited by the inability handle unknown inputs. Finally, the results of Huffpost (2017) and MAS- SIVE, which exhibit a high domain gap, are re- ported in Table 5. The result indicates that UniDA methods consistently display superior performance in most cases. However, the divergence between the approaches is relatively small compared to Amazon. UniOT demonstrates the best perfor- mance all datasets, with OVANet's slightly lower performance. AdSPT demonstrates marginally bet- ter performance than OVANet in Huffpost (2017),
5.3 Impact Threshold Values
The selection of the threshold value considcrably influences the performance of CDA methods. In order probe the impact of the threshold values the performance, carry out analysis whereby different threshold values are applied measure the performance of the methods.
The results are demonstrated in Figure 4. In cases of low or moderate adaptation complexity, such CLINC-150 and Huffpost (2013, 2014, 2015, 201 6), CDA methods demonstrate the po- tential to outperform UniDA methods when pro- vided an appropriate threshold. However, as the adaptation complexity intensifies. such as Huffpost (2017), MASSIVE, and Amazon, UniDA meth- ods outperform CDA methods regardless of the selected threshold. These observations align seam-

---

A427
UN
ASSPT
Uriot
U107
ACSPT
100
100
100
80
30
80
80
H-score
40
40
40
40
20
20
20-
20-
8.0
1.0
0.0
0.0
8.0
Threshold
Threshold
Threshold
Threshold
(a) CLINC-150
(b) Huffpast (2012 -> 2013)
(c) Huffpast (2012 - 2014)
(d) Huffpost (2012 - 2015)
Ad577
UN
AZ3FT
UnioT
ATD
U1.T
AcdFT
LIIOT
1.00
100
100
100
80
80
80
80
H-scome
40
40
40
40
20
8.0
I.
8.0
0.2
0.4
8.0
0.4
0.6
0.8
1.0
8.0
1.0
Threshold
Threshold
Threshold
Threshald
(c) Huffpost (2012 + 2016)
(f) Huffpost (2012 + 2017)
(g)
MASSIVE
(h) Amazon
Figure 4: H-score performance with different threshold values. Results of UniDA methods are visualized horizontal line for comparison.
lessly with the findings from Section 4.3 that under- score the proficiency of UniDA methods in manag- ing challenging adaptation scenarios. Additionally, it should be noted that determining the optimal threshold is particularly challenging in the absence of supervision from the target domain. Therefore, the best performance should be considered upper- bound of the CDA methods.
You et al. (2019) quantifies sample-level transfer- ability by using of uncertainty and domain similar- ity. Following the work, Fu al. (2020) calibrates multiple uncertainty measures to handle such an issue. Saito and Saenko (2021) apply a one-vs- all classifier to minimize inter-class distance and classify unknown classes. More recently, Chang et al. (2022) applied Optimal Transport and further expanded the task to discovering private classes, Other recent works focus utilizing mutually nearest neighbor samples (Chen et al., 2022a,c) leveraging source prototypes with target samples (Chen et al., 2022b; Kundu al., 2022). Despite the practicality of UniDA, its application in the NLP domain has barely explored.
6 Related Work
6.1 Domain Adaptation
The studies in the field of DA in NLP primar- ily assumes a closed-set environment, which the source and the target domain share the same la- bel space. CDA research predominantly concen- trated on learning domain invariant features (Blitzer et al., 2006; Pan et al., 2010; Ben-David et al., 2020; Ganin and Lempitsky, 2015; Du et al., 2020) for effective adaptation. With the advent of pre- trained language models (PLMs), CDA methods have evolved to effectively leverage the capabili- ties of PLMs. Techniques such as masked language modeling (Karouzos et al., 2021) or soft-prompt with adversarial training (Wu and Shi, 2022) have shown promising results. However, the closed-set assumption has a fundamental drawback as it may leave the models vulnerable when exposed to data from an unknown class.
6.2 Out-of-Distribution Detection
The early exploration of OOD detection focused on training supervised detectors (Dhamija et al., 2018; Lee et al., 2018a; Jiang et al., 2018). However, since obtaining labeled OOD samples is imprac- tical, recent OOD detection research has shifted towards unsupervised methods, such as generat- ing pseudo-OOD data (Chen and Yu, 2021; Zheng et al., 2020), utilizing self self-supervised learning (Moon et al., 2021; Manolache et al., 2021; Li et al., 2021; Zeng et al., 2021; Zhan et al., 2021; Cho et al., 2022), and measuring uncertainty through scoring functions for input instances (Hendrycks and Gimpel, 2017: Lee et al., 2018b; Liu et al., 2020; Tack et al., 2020). While these methods have shown effectiveness, OOD detection is limited
To mitigate such issue, a new line of work named UniDA (You et al., 2019) was proposed which us- sumes no prior knowledge about the target domain.

---

that it does not offer opportunities for adaptation.
concepts such as domain shifts and category shifts in generative tasks. However, in light of recent ad- vancements in the generative capabilities of models, handling distributional shifts in generative tasks is indubitably an essential problem that needs to be addressed in our future work.
7 Conclusion and Future Work
In this study, we present a testbed for evaluating UniDA in the field of NLP. The testhed is designed to exhibit various levels of domain and category gaps through different datasets. Two novel metrics, PDR and DDS, were proposed which can mea- sure the degree of domain and category gap, re- spectively. We assessed UniDA methods and the heuristic combination of CDA and OOD detection in our proposed testbed. Experimental results show that UniDA methods, initially designed for the vi- sion domain, can be effectively transferred to NLP. Additionally, CDA methods, which are not fully optimized in UniDA scenario, produce comparable results in certain circumstances.
A ckknowledgement
This work was mainly supported by SNU-NAVER Hyperscale Al Center and partly supported by In- stitute of Information & communications Technol- ogy Planning & Evaluation (ITPP) grant funded by the Korea government (MSIT) (N..2020-0-01333, Artificial Intelligence Graduate School Program (Hanyang University), Hub (Artificial Intel- cial Intelligence Innovation ligence Institute, Seoul National University)] and Korea Evaluation Institute of Industrial Technol- ogy (KEIT) grant funded by the Korea governmet (MOTIE).
Artifi-
Recent trends in NLP focus Large Language Models (LLMs) of their significant generalization abilities. However, the robustness of LLMs from the perspective of UniDA remains uncertain. As part of our future work, we assess the performance and the capabilities of LLMs from a UniDA view- point.
References
Charu C. Aggarwal. 2017. An Introduction to Outlier Analysis, pages 1-34. Springer International Publish- ing, Cham.
Limitations
Limited coverage of the evaluated model sizes The evaluation was conducted only with models of limited size. Moreover, there is a lack of zero- shot and few-shot evaluations for large language models (LLMs) that have recently emerged with remarkable generalization capabilities. The evalu- ation of LLMs is currently being considered as a top priority for our future work, and based on pre- liminary experiments, the results were somewhat unsatisfactory compared to small models with ba- sic tuning for classification performance. In this regard, recent research that evaluated LLMs for classification problems (such GLUE) also re- ported that the performance is not yet comparable to task-specifically tuned models. Considering the limitations of LLMs in terms of their massive re- source usage and the fact that tuning small models still outperforms them in a task-specific manner, the findings from this study are still considered highly valuable in the NLP community.
Eyal Ben-David, Carmel Rabinovitz, and Roi Reichart. 2020. Perl: Pivot-based domain adaptation for pre-trained deep contextualized embedding models. Transactions of the Association for Computational Linguistics, 8:504-521.
John Blitzer, Ryan McDonald, and Fernando Pereira. 2006. Domain adaptation with structural correspon- dence learning. In Proceedings of the 2006 Con- ference on Empirical Methods in Natural Language Processing, pages 120-128, Sydney, Australia. Asso- ciation for Computational Linguistics.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Voss, Gretchen Krueger, Tom Henighan, Rewon Child. Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray. Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Tlya Sutskever, and Dario Amodei. 2020. Language models are few-shot leamner. In Ad- vances Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Limited scope of the tasks Our proposed testbed is restricted to text classification tasks only. The majority of existing research on DA and OOD also focuses on classification. This selective task prefer- ence is primarily due to the challenge of defining
Wanxing Chang, Ye Shi, Hoang Duong Tuan, and Jingya Wang. 2022, Unified optimal transport frame- work for universal domain adaptation. In Advances in Neural Information Processing Systems.

---

Derek Chen and Zhou Yu. 2021. GOLD; Improving out-of-scope detection in dialogues using data aug- mentation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 429-442, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
and Noah Fiedel. 2022. Palm: Scaling language mod- eling with pathways.
Jacob Devlin, Ming-Was Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Liang Chen, Qianjin Du, Yihang Lou, Jianzhong He, Tao Bai, and Minghua Deng. 2022a. Mutual near- est neighbor contrast and hybrid prototype self- training for universal domain adaptation. Proceed- ings of the AAAI Conference on Artificial Intelligence, 3666):6248-6257.
Akshay Raj Dhamija, Manuel Güither, and Terrance Boult. 2018. Reducing network agnostophobia. Ad- vances in Neural Information Processing Systems, 31.
Liang Chen, Yihang Lou, Jianzhong He, Tao Bai, and Minghua Deng. 2022b. Evidential neighborhood contrastive learning for universal domain adaptation. Proceedings of the AAAI Conference on Artificial Intelligence, (6(6))6258-6222) 6267.
Chunning Du, Haifeng Sun, Jingyu Wang. Qi Qi, and Jianxin Liao. 2020. Adversarial and domain-aware bert for cross-domain sentiment analysis. In Proceed- ings of the 58th annual meeting of the Association for Computational Linguistics, pages 4019-4028.
Liang Chen, Yihang Lou, Jianzhong He, Tao Bai, and Minghua Deng. 2022c. Geometric anchor correspon- dence mining with uncertainty modeling for universal domain adaptation. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16112-16122
Jack FitzGerald, Christopher Hench, Charith Peris, Scott Mackie, Kay Rottmann, Ana Sanchez, Aaron Nash, Liam Urbach, Vishesh Kakarala, Richa Singh, Swetha Ranganath, Laurie Crist, Misha Britan, Wouter Leeuwis, Gokhan Tur, and Prem Natara- jan. 2022. Massive: Im-example multilin- gual natural language understanding dataset with 51 sypoogicalll-divert languages.
Hyunsoo Cho, Choonghyun Park, Jaewook Kang, Kang Min Yoo, Tacuk Kim, and Sang-goo Lee. 2022. Enhancing out-of -iittiibbtti detection in natural language understanding via implicit layer ensem- ble. In Findings of the Association for Computa- tional Linguistics: EMNLP 2022, pages 783-798, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Bo Fu, Zhangjic Cao, Mingsheng Long, and Jianmin Wang. 2020. Learning to detect open classes for uni- versal domain adaptation. In European Conference on Computer Vision.
Hyunsoo Cho, Jinseok Seol, and Sang-goo Lee. 2021. Masked contrastive learning for anomaly detec- tion. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pages 1434-1441. International Joint Conferences on Artificial Intelligence Organization. Main Track.
Yaroslav Ganin and Victor Lempitsky. 2015. Unsu- pervised domain adaptation by backpropagation. In Proceedings of the 32nd International Conference on International Conference on Machine Learning- Volume 37, ICML'15, page 1 1180-1189. JMLR.org.
Aakanksha Chowdhery, Sharan Narang. Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko. Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vin- odkumar Prabhakaran, Emily Reif, Nan Du Ben Hutchinson, Reiner Pope. James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengeheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, An- drew M. Dai, Thanunalayan Sankaranarayana Pil- lai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Sacta, Mark Diaz. Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellster, Douglas Eck, Jeff Dean, Slay Petrov,
Yaroslav Ganin, E. Ustinova, Hana Ajakan, Pascal Germain, H. Larochelle, François Laviolette, Mario Marchand, and Victor S. Lempitsky, 2016a. Domain- adversarial training of neural networks. In Journal of machine learning research.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Lavio- lette, Mario Marchand, and Victor Lempitsky. 2016b. Domain- ad aevesaaiial training of neural networks. The journal of machine learning research, 17(1 1711)200 2030.
Dan Hendrycks and Kevin Gimpel. 2017. A baseline for detecting misclassified and out- distriibittiinnoonn exam- ples neural networks. In International Conference on Learning Representations.
Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan, and Dawn Song. 2020. Pretrained transformers improve out-of of-iiiiiiiuttion

---

robustness. In Proceedings of the 58th Anmual Meet- ing of the Association for Computational Linguistics, pages 2744-2751, Online. Association for Computa- tional Linguisties.
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. 2020. Energy-based out- out-of-distributi distribbution detec- tion. In Advances in Neural Information Processing Systems, volume 33, pages 21464-21475. Curran As- sociates, Inc.
Dan Hendrycks, Mantas Mazeika, and Thomas Diet- terich. 2019. Deep anomaly detection with outlier exposure. In International Conference on Learning Representations.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Confer- ence on Learning Representations.
Andrei Manolache, Florin Brad, and Elena Burceanu. 2021. DATE: Detecting anomalies in text via self- supervision of transformers. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 267-277, Online. Association for Computational Linguisties.
Heinrich Jiang, Been Kim, Melody Guan, and Maya Gupta. 2018. To trust or not to trust a classifier. Advances in neural information processing systems, 31.
Constantinos F. Karouzos, Georgios Paraskevopoulos, and Alexandros Potamianos. 2021. Udalm: Unsuper- vised domain adaptation through language modeling. ArXiv, abx/2104.07078.
John Miller, Karl Krauth, Benjamin Recht, and Ludwig Schmidt. 2020. The effect of natural distribution shift on question answering models. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 6905-6918 PMLR.
Phillip Keung, Yichao Lu, György Szarvas, and Noah A. Smith. 2020. The multilingual Amazon reviews cor-
pus. In Proceedings of the 2020 Conference Empirical Methods Natural Language Processing (EMNLP),, pages 4563-4568, Online, Association for Computational Linguistics.
Rishabh Misra. 2022. News category dataset. arXiv preprint arXiv:2209.
Jogendra Nath Kundu, Suvaansh Bhambri, Akshay Kulkarni, Hiran Sarkar, Varun Jampani, and R. Venkatesh Babu. 2022. Subsidiary prototype alignment for universal domain adaptation. In Ad- vances in Neural Information Processing Systems (NeurIPS).
Seung Jun Moon, Sangwoo Mo. Kimin Lee, Jaeho Lee. and Jinwoo Shin. 2021. Masker: Masked keyword regularization for reliable text classification. Pro- ceedings of the AAAI Conference on Artificial Intelli- gence, 35( (15): 1377--1356..
OpenAI. 2023. Gpt-4 technical report.
Stefan Larson, Anish Mahendran, Joseph Peper, Christopher Clarke, Andrew Lec, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjia Tang, and Jason Mars. 2019. An evaluation dataset for intent classification and out-of- scope prediction. In Proceedings of the 2019 Confer- ence on Empirical Methods Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-LJCNL pages 131 1-1316, Hong Kong, China, Association for Computational Linguistics.
Sinno Jialin Pan, Xiaochuan Ni, Jian-Tao Sun, Qiang Yang, and Zheng Chen. 2010. Cross-domain senti- ment classification via spectral feature alignment. Proceedings of the 19th International Conference on World Wide Web, wWw '10 page 751-760, New York, NY, USA. Association for Computing Machin- ery.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32.
Kimin Lee, Honglak Lec, Kibok Lee, and Jinwoo Shin. 2018a. Training confidence-callbraaeed classifiers for detecting -ui-o--disttibution samples. Interna- tional Conference on Learning Representations.
Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Be- havioral testing of NLP models with CheckList. In Proceedings of the 58th Annual Meeting the Asso- ciation for Computational Linguistics, pages 4902- 4912, Online. Association for Computational Lin- guistics.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. 2018b. A simple unified framework for detecting of-distribution samples and adversarial attacks. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc.
Tian Li, Xiang Chen, Shanghang Zhang, Zhen Dong. and Kurt Keutzer. 2021. Cross-domain sentiment classification with contrastive learning and mutual information maximization. In ICASSP 2021- 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8203-8207.
Kuniaki Saito and Kate Saenko. 2021. Ovanet: One-vs- all network for universal domain adaptation. arXiv preprint arxiv:2104.03344
Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin, 2020. Csi: Novelty detection via contrastive

---

learning on distributionally shifted instances. Ad- vances in Neural Information Processing Systems, volume 33, pages 11839-11852. Curran Associates, Inc.
Yinhe Zheng, Guanyi Chen, and Minlie Huang. 2020. Out-of-domain detection for natural language un- derstanding in dialog systems. IEFF/ACM Trans- actions on Audio, Speech, and Language Processing, 28:1 1209.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models.
A
Full Experimental Results
A.1 UniDA Results
Table is the full results of UniDA methods our proposed testbed. Baseline methods such as UAN (You et al., 2019) and CMU (Fu et al., 2020) are included in the results. We can observe that UniDA methods do not always retrain the same level of applicability in NLP. Specifically, UAN and CMU utilize a fixed threshold defined in the vision do- main. While CMU remains fully compatible in the NLP domain, UAN struggles to apply effectively, as it fails to detect unknown samples.
Jindong Wang, Xixu Hu, Wenxin Hou, Hao Chen, Runkai Zheng. Yidong Wang, Linyi Yang. Haojun Huang. Wei Ye, Xiubo Geng, Binxin Jiao, Yue Zhang. and Xing Xie. 2023. On the robustness of chatgpt: An adversarial and out-of-distribution perspective. arXiv preprint arXiv: 1209.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- iez. Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38- Online. Association for Computational Linguistics.
A.2 CDA Results
In this section, we demonstrate the experimental results of CDA methods with two additional scor- ing functions: cosine similarity and Mahalanobis distance. The threshold value was selected based on the score from the scoring functions, using the same approach as the main experiment. Also, we report the results of DANN (Ganin et al., 6a) and source-only fine-tuning which was left out from the main experiment. In some cases, source- only fine- tuning outperforms other adaptation meth- ods, which is also observed in the vision domain (You et al., 2019).
Hui Wu and Xiaodong Shi. 2022. Adversarial soft prompt tuning for cross-domain sentiment analysis.
In Proceedings of the 60th Annual Meeting the Association for Computational Linguistics (Volume
1: Long Papers), pages 2438-2447, Dublin, Ireland. Association for Computational Linguisties.
Kaichao You, Mingsheng Long, Zhangjee Cao, Jian- min Wang, and Michael I. Jordan. 2019. Universal domain adaptation. In The IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR).
Cosine Similarity (Tack et al., 2020) calcu- lates the cosine similarity score between the test input and the train input. The score is selected as the cosine similarity between the input and the nearest neighbor. The results are reported in Table 8.
Zhiyuan Zeng, Keqing He, Yuanmeng Yan, Zijun Liu, Yanan Wu, Hong Xu, Huixing Jiang, and Weiran Xu. 2021. Modeling discriminative representations for out-of-domain detection with supervised contrastive learning. In Proceedings of the 59th Anmual Meet- ing of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natu- ral Language Processing (Volume 2: Short Papers), pages 870-878, Online. Association for Computa- tional Linguistics.
2. Mahalanobis Distance (Lee et al., 2018b) is the distance of the test sample to each class distribution. The representation is assumed to follow the multivariate normal distributions. The distance between the nearest class distri- bution is used as the score. The results are demonstrated in Table 9
Li- LinMing Zhan, Haowen Liang. Bo Liu, Lu Fan, Xiao- Ming Wu, and Albert Y.S. Lam. 2021. Out-of-scope intent detection with self-supervision and discrimi- native training. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3521-3532, Online. Association for Computational Linguistics.
Additionally, the full experimental results of MSP thresholding are presented in Table 10.

---

Methods
BERT
DANN
UDALM
AdSPT
the average results of three different runs.
Hufposst (2013)
5e.5
5e:5
le-4
5e-5
Table 11 is the results of the experiments. Due the changes in the class set to be predicted, the task difficulty varies, resulting in differences in abso- lute performances. However, when comparing the relative performance between different methods, le-5
Hullpost (2014)
Se-5
5e-5
5e-5
Hullpost (2015)
Se-5
Se-5
le-4
5e-5
Huffpost (2016)
5e-5
5e-5
le-4
5e-5
Huffpost (2017)
5e-5
le.4
5e-5
CLINC-150
le.4
5e-4
Se-4
MASSIVE
5c-5
5c-5
5c-5
5e-5
we can observe that they exhibit consistent trends regardless of the class split.
Amazon
le-5
5e-6
5e-5
le-5
UniDA Methods
UAN
CMU
OVANet
UniOT
D Receiver Operating Characteristic (ROC) Curve
Huffpost (2013)
5e-5
5e-5
5e-5
le-4
Huffosst (2014)
5e:5
5e-5
5e-5
le-4
Hulfpost (2015)
le.4
le.4
Se-5
5e-5
Hullpost (2016)
le-4
Se-5
5e-5
le-4
To measure Distinction Difficulty Score (DDS), we calculated the AUROC and subtracted from 1. Fig- ure 5 is the ROC curve of discerning unknown in- puts from the transferable inputs for our proposed datasets. The closer the ROC curve is to the upper- left corner, indicates that is easier to distinguish between unknown and transferable inputs.
Hutfpost (2017)
le-4
Se-5
5e-5
le44
CLINC- 150
5e-6
le:5
5e-6
5e-5
MASSIVE
5e-5
5e-5
Se-5
le-4
Amazon
5e-5
Se-5
5e-5
5e-5
Table 6: Learning rates for each methods. The learning rate was selected from Sc-4, le-4, 5e-5, le-5, and le-6 with the best evaluation performance from the source domain.
B Implementation Details
For the experiments, we adopt 12-layer pre- trained language model bert-base-uncased (Devlin et al., 2019) as the backbone all the methods. We utilized the [CLS] representation as the input feature. Adam W optimizer (Loshchilov and Hut- ter, 2019) was used for all the experiments with batch size of 32. We selected the best learning rate among 5e-4, le-4, 5c-5, 1c-5, and 5e-6. The learning rate for each method reported in Table 6. The model was trained for 10 epochs with an early stopping on the accuracy of the source do- main's evaluation set. All the experiments were implemented with Pytorch (Paszke et al., 2019) and Huggingface Transformers library (Wolf et al., 2020). The experiments take an hour single Tesla V100 GPU.
C Ablation on Different Class Splits
For the main experiment, we utilized class names as the criterion to implement the category gap. How- ever, this may only show the specific scenario of the category gap. To provide more comprehen- sive analysis, also report the results when the class set is randomly split. We utilized CLINC-150 and MASSIVE dataset for the ablation study, and MSP thresholding was applied for CDA methods. We conducted three experiments, each with dif- ferent class split, and for every split, we reported

---

Dataset
Huffpost (2012
2013) (3/4/4)
Huffpost (2012 + 2014) (3/4/4)
Method
accc
accc,
H-score
accc
accč;
H-score
UAN
73.58 +5.76
0.00
+0.00
0.00
=0.00
36.38 +3.04
0.00 +0.00
0.00
=0.00
CMU
58.18 +1.88
26.89
+4.29
36.76
+4.22
33.13
+3.35
39.98 +3.71
36.02
1146
OVANet
65.11 +0.60
24.91 +6.75
35.64
+6.70
45.85
+2.62
33.96 +6.78
38.56 +3.95
UniOT
53.76 11.10
65.86 13.57
59.14
11.14
33.49
14.79
71.44 16.19
45.23 13.60
Dataset
Huffpost (2012 > 2015) (3 / / 4)
Huffpost (2012 -> 2016) (3 / 4 4)
Method
accc
accc,
H-score
acCc
accc.
H-score
UAN
34.17 +6.00
0.00 +0.00
0.00 =0.00
42.53
16652
0.00 +0.00
0.00 =0.00
CMU
39.57 13.54
17.98
16.30
24.02
15.57
41.80 12.08
30.09 13.51
34.84 11.97
OVANet
43.91
12.09
35.70
14.41
39.21
12.43
44.30
12.89
33.49 14.79
38.03
13.79
UniOT
27.16 +3.40
75.50 +2.68
39.82
+3.39
33.09 +3.62
69.27 +3.83
44.60 +2.85
Dataset
Huffppsst (2012
2017) (3 / 11/4/4 4)
CLINC- 150 (4 3 //3)
Method
accc
accc,
H-score
accc
accC;
H-score
UAN
22.74 110.63
0.00
10.00
0.00 10.00
48.99
114.49
0.00 10.00
0.00
10.00
CMU
34.87 +3.14
55.79
+2.71
42.86 +2.72
60.70 +1.02
36.89 +1.52
45.87 +1.09
OVANet
48.37 +4.12
35.03
+3.60
40.46 +2.32
83.49
+0.62
31.24 +1.70
45.45 +1.88
UniOT
31.57 +1.93
75.85 +9.63
44.36 +1.27
64.14 19.14
77.36 +3.94
69.88 +6.11
Dataset
MASSIVE 58/5/5)
Amazon /11/10/10 10)
Method
acec
accc,
H-score
accc
accc,
H-score
UAN
13.00 +7.18
0.00
+0.00
0.00 =0.00
35.40 +3.44
0.00
+0.00
0.00 =0.00
CMU
37.38 +1.59
10.33
+3.06
16.03 13.84
44.30 40.90
39.95
19.67
41.48 45.10
OVANet
59.04
+0.69
39.25
+2.14
47.12 +1.77
44.50 +0.84
42.49
+3.95
43.38 +1.83
UniOT
44.01 13.52
79.19
14.14
56.52 13.49
38.60 11.08
70.98 17.76
49.90 12.08
Table 7: Experimental results of UniDA methods in the proposed testhed. For each dataset, the best method with the highest H-score bold and second-best method is underlined.

---

Dataset
Huffpost (2012
2013) (3/4/4
Huffpost (2012 -> 2014) (3 /
Method
accc
accç,
H-score
accc
accc,
H-score
BERT
64.06 +2.13
28.02 +3.03
38.86
22.63
40.63 #3.55
37.41 #4.81
38.65 +1.34
DANN
58.92 11.62
1.69 10.57
3.28 11.07
33.08
12.44
2.01 10.82
3.76 11.48
UDALM
60.12 +2.56
23.94 +3.80
34.12 +4.00
37.97 +1.69
29.27 +6.15
32.84 +4.22
AdSPT
62.71 +0.88
10.89 +3.04
18.41
#4.38
39.57 33.8
23.88 +3.44
29.74
#3.59
Dataset
Huffpost (2012
2015) (3 / 4)
Huffpost (2012 - 2016) (3 / 44)
Method
accc
accç,
H-score
accc
accc,
H-score
BERT
40.75 12.12
30.90
+1.55
35.13
1153
47.07
1246
26.15 43.73
33.42 12.43
DANN
31.71 12.49
3.39 +0.92
6.11
+1.57
36.68
3.86
2.16
+0.80
4.05 11.44
UDALM
37.06 +1.74
24.79
+4.83
29.59
+3.89
+1.52 44.10
23.26
+2.48
30.38 +2.02
AdSPT
39.92 +4.75
15.01 +3.85
21.74
+4.84
41.44 3.66
12.66 +3.31
19.14 #3.97
Dataset
Huffpost (2012 + 2017) (3 (3/4/4)
CLINC-150 (4/3/3)
Method
accc
acco,
H-score
accc
aCcc,
H-score
BERT
48.84 13.16
29.52
14.67
36.62
13.91
74.53
13.02
71.37
16.54
72.69
12.48
DANN
31.30 +2.34
9.32
+166
14.31
#2.02
60.21 +9.33
49.65
+14.72
52.74
110115
UDALM
41.08
13.92
29.10
17.01
33.48
13.83
75.99
13.63
78.11
15.41
76.90
12.35
AdSPT
43.02 +0.74
31.07 +4.77
35.93
+3.37
42.94 +5.59
6.96
+3.44
11.57 +4.59
Dataset
MASSIVE 58/)
Amazon (11 10 10)
Method
accc
acco,
H-score
accc
accc,
H-score
BERT
51.21 13.62
62.96
12.06
56.44
12.69
44.88
12.10
14.49
11.25
21.90 11.68
DANN
41.55 19.00
13.74
43.52
20.62
+5.01
44.58
10.97
19.61
+1.08
27.21 10.93
UDALM
65.22 +2.67
64.25
+2.17
64.67
10.76
46.65
10.47
13.58
+2.23
20.96
1:2.79
AdSPT
36.78 +2.65
59.87
+13.47
45.32
+6.20
30.26 +1.45
6.45 +4.55
10.15 +5.65
Table 8: Experimental results of CDA methods with cosine similarity the scoring function. For each dataset, the best method with highest. H-score is in bold and the second-best. method underlined.

---

Datasct
Huffpost (2012 - 2013) (3 (3/4/4) 4)
Huffpoost (2012 + 2014) (3/ 1//4/4
Method
acec
acccc
H-score
0.CC
accic,
H-score
BERT
10.40 +12.11
35.69
+3.72
13.78
+2.38
11.28 +8.86
42,44 +2.47
16.41 +9.26
DANN
18.55 19.65
1.50 10.30
2.69 10.62
16.78
+7.67
1.48 +0.67
2.70 #1.22
UDALM
7.39 10.12
27.73
14.25
9.26 110.18
4.92
14.14
34.04 15.66
8.15 1655
AdSPT
18.07 10066
7.70 :0.36
10.21 +2.49
17.29 +8.06
18.80 +3.47
16.87 +5.39
Dataset
Huffpost (2012 + 2015) (3 4 4)
Huffpost (2012 -> 2016) (3 4//4/
Method
accc
accc,
H-score
aCCC
accc,
H-score
BERT
11.28 +11.10
37.71 14.18
15.38
111.27
10.34 19.85
31.43 15.76
14.52
110.93
DANN
20.09 =5.97
2.93
+1.23
4.88
+1.59
20.79
110.71
1.92 40.32
3.44 =0.41
UDALM
4.19 12.98
31.19
16.15
7.06
14.84
6.23
25.09
31.52 15.19
9.49 6674
AdSPT
19.20 +7.31
14.55 +3.02
16.30
+4.47
17.78 19.51
9.73 44.76
12.10 16.48
Dataset
Huffppost (2012
2017) (3 /4)
CLINC-150 /4/3/3)
Method
acec
accC,
H-score
aCCC
000G
H-score
BERT
10.31 +13.20
34.04
+5.04
13.10
+13.18
16.67 +21448
72.56
45.44
21.87 +:5.69
DANN
16.79 +0.94
7.48
+1.62
10.29
+1.63
1.87 #1.66
41.89
-20.06
3.25 =1.89
UDALM
7.65 +5.29
30.08
+8.33
10.93 16.62
5.18 +8.80
80.15 43.95
8.73 +14400
AdSPT
19.77 18.68
30.93 13.52
23.30
17.35
6.59 -7.97
29.85 -29.28
7.41
Dataset
MASSIVE (8 5/)))
Amazon (11 10 10)
Method
accc
accE,
H-score
acec
0000,
H-score
BERT
5.62 +9.41
68.53 +4.54
9.07 +14.7
4.66 +2.59
15.97 +1.59
6.85 #3.45
DANN
4.59 +3.59
9.73 +3.60
5.26 +2.70
8.62 +7.31
17.74 +3.09
9.75 +3.92
UDALM
2.96 +4190
62.80
+5.02
5.17 18.36
2.68 +2.80
19.01 45.49
4.48 4.22
AdSPT
5.70 +6.41
54.61 +12.44
9.60 100.75
4.63 +4.66
9.38 +6.08
5.61 +5.65
Table 9: Experimental results of CDA methods with Mahalanobis distance as the scoring function. For each dataset, the best method with the highest H-score is in bold and the second-hest method is underlined.

---

Dataset
Huffpost (2012 2013) (3/4/4)
Huffpost (2012 - 2014) (3 44/4)
Method
accc
accc,
H-score
accc
accc,
H-score
BERT
51.47 23.72
72.07 110.30
59.64
13.28
26.71
11.99
78.98 18.23
39.75 +1.51
DANN
49.35 +6.58
77.79 +25.05
57.83
+6.49
14.94
+3.26
95.24
+0.70
25.71 #148
UDALM
52.74 +2.92
58.25 +6.03
55.15 +2.37
29.90
+1.89
50.00 +17.49
36.14 +5.07
AdSPT
55.05 #2.11
80.66 +2.99
65.38 +1.00
29.64 +4.51
80.03 +6.09
42.93 #3.71
Dataset
Huffpost (2012 - 2015) (3 4 4)
Huffpost (2012
2016) (3 4/44)
Method
aCCC
accec
H-score
aCCc
accč:
H-score
BERT
27.35 +2.61
77.13 +:220
40.30
+:2.0
33.12
+185
63.65
+3.63
43.53
1.96
DANN
16.48 +7.90
89.93
+1151
26.81 +9.33
11.23 +4.13
95.41
+1.23
19.89
+0.52
UDALM
25.15 +2.60
61.79 +8.88
35,56 13.15
29. 14
+1.9
60.87
16.87
39.28
1.44
AdSPT
29.08 15.39
73.04 112.71
40.78 13.42
28.84 13.57
82.52
14.22
42.55
13.27
Datasct
Huffpost (2012 - 2017) (3 1/4/4/ 4)
CLINC-150 (4 /
Method
aCcc
accc,
H-score
accc
accc,
H-score
BERT
33.07 13.90
65.82 17.47
43.64 11.71
64.69
13.68
72.08 19.82
67.82
14.15
DANN
22.84 =2.60
82.46
+-139
35.66
#2.85
48.02 +4.93
67.48 119.8
54.90
19.17
UDALM
32.85 11.05
43.08 16.76
37.08
12.82
74.92
16.11
69.91 113.98
71.28
14.62
AdSPT
35.74 23.33
53.04 16.02
41.36 +5.01
3.96 +:217
98.69 10.36
7.55
13.95
Dataset
MASSIVE (8/5/5)
Amazon (11 / 10 10)
Method
accc
accc,
H-score
accc
accc,
H-score
BERT
30.04 -2.74
68.95
+5.53
41.79 +:3.0
44.98
+2.32
12.66 +1.47
19.70
+1.69
DANN
20.37 +11.32
88.69 +5.27
31.77 +13.32
45.85
+0.49
12.52
+0.71
19.66 +0.87
UDALM
38.33 +4.88
41.29
+15.14
39.28 +9.60
47.69
+1.47
15.10
+2.30
22.85 2266
AdSPT
36.78 12.65
59.87 113.47
45.32 16.20
30.26
13.52
30.93
126.14
27.12 +1100
Table 10: Experimental results of CDA methods with Maximum Softmax Probability the scoring function. For each dataset, the best method with the highest H-score is in bold and the second-ber method is underlined.
Dataset
CLINC-150 34/3/3)
MASSIVE 5/))
Runs
2
3
3
BERT
44.59 16.01
48.08
+2.90
52.25
+3.54
51.68
41.62
46.68
+1.90
51.56
+0.81
DANN
47.91 19.96
53.23
17.92
60.42
17.00
47.16
13.79
31.39
13.93
45.21
18.93
UDALM
49.17 18.75
55.04
15.31
52.42
18.20
55.52
13.00
15.88 42.34
51,15 13.62
AdSPT
31.81 +8.63
48.62
+1.85
+5.22 46.20
45.41
+9.04
38.61
+3.90
46.95 +5.22
UAN
0.00 10.00
0.00 10.00
0.00 10.00
0.00 10.00
0.00
10.00
0.00
10.00
CMU
40.32 +2.32
38.79 +2.19
40.92
11.77
21.46 +1.62
18.84 +5.26
28.33 +4.29
OVANet
41.77 11.90
35.89 13.86
29.15
12.17
63.91 13.41
54.31 10.63
39.85 13.74
UniOT
50.94 +7.20
61.66 +1.70
56.51
+3.53
62.07 +2289
55.11 +2.31
63.59 +3.15
Table 11: H-score results of UniDA methods in CLINC-150 and MASSIVE with different class splits. The best method with the highest H-score is in bold and the second-best method underlined.

---

Thue Positive Rate
Thus Positive Rate
True Positive
Pastive
0.5
0.5
0.5
10C Givee (ACC 6.01
RDC
0.0
Randon
0.0
Random (60I 6.5)
0.0
Randon (hil
0.0
1.0
1.0
Ralse
Huffpost (2013)
(b) Huffpoost (2014)
Huffpost (2015)
(d) Huffpost (2016)
1.0
True Positive Rate
Thus Postive Rate
Positive
True
ROC. Gine (AUC 0.68)
RGC aive (AU 6.85
RDC orve (IX - 3.6 Random (MIC 0.51
ROC cine (AUC 0.641 Random (uic = 0.)
Randem (AUC = 8.5)
0.0
Randam MMAC 2.5
1.0
0.0
0.0
1.0
1.0
Fabe Positive Rate
Folse Positive Rate
False Positive Rate
Fabs Positive Rate
(e) Huffpost (2017)
(f) CLINC-150
(g) MASSIVE
(h) Amazon
Figure 5: ROC curve of discerning unknown samples from the rransferable samples. The closer the ROC curve is the upper-left corner, it becomes casier to distinguish between the them.


Reviews:
[{"review_id": "H1BhEMG47T", "reviewer": "Reviewer_wuhF", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for dedicating your valuable time and effort to reviewing our paper. Our responses to the comments are as follows.\n\n\n\n**W1. Limited Novelty**\n\nThe reviewer has pointed out that our work is a simple migration of methods from the vision domain, \nbut we would like to strongly argue that the main contribution of our work is to address the importance of relatively unexplored Universal Domain Adaptation (UniDA) in the NLP field and serve as the cornerstone of pioneering and setting a solid environment for future research.\nSpecifically, we made efforts to construct a benchmark including various types of shifts in NLP (e.g., sentiment, temporal shifts) and different difficulty levels. (Section 3.3)\nThe levels of difficulty were quantitatively measured by two novel metrics (PDR and DDS) which measures the degree of domain and category gap. (Section 3.1)\n\nAs the reviewer pointed out, existing methods from the computer vision domain were migrated and evaluated on our benchmark.\nHowever, we also made efforts to evaluate state-of-the-art methods from the NLP domain for comparative analysis between existing methods.\nBy verifying the effectiveness of prior studies within this benchmark, we believe we have laid a robust foundation for future UniDA research.\nWe humbly urge that our contribution involves efforts beyond simple migration, aiming to integrate and solidify the UniDA setting in the NLP field. \nYour thoughtful consideration of this aspect would be greatly appreciated.\n\n\n\n**W2. Limited to Text Classification**\n\nWe agree with the reviewer’s point and are fully aware that our work is concentrated only on text classification tasks.\nThe fact that existing UniDA literature in the vision domain is predominantly focused on the classification task has significantly influenced the scope of our research.\nAs our work is the first to introduce the concept of UniDA to the NLP field, we deemed it appropriate to follow existing works and begin the research within the classification tasks.\nAs mentioned in the Limitation Section, the selective task preference is also due to the challenge of defining concepts such as domain and category shifts in tasks such as text generation.\nWe advocate for gradually expanding UniDA’s scope to encompass advanced tasks such as text generation.\nDespite the limitation, we believe our paper provides comprehensive empirical analysis exclusively within the text classification task, offering diverse insights.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The testbed includes datasets divided according to difficulty levels, evaluated by two new evaluation metrics. By dividing the datasets based on the degree of difficulty, the capabilities of each method can be better distinguished. \n2. The experiment is relatively complete, with many types of datasets and a comprehensive comparison of different methods. The effectiveness of the UniDA methods can be demonstrated.", "reasons_to_reject": "The proposed approach simply migrates existing methods from computer vision to solve the generalization problem in the field of natural language processing, limiting the novelty of this paper. Additionally, it is expected that experiments will be conducted to test the generalization ability in more NLP applications, beyond text classification.", "questions_to_authors": "", "paper_topic_and_main_contributions": "While UniDA has led to significant progress in computer vision, its application to language input still needs to be explored, despite its feasibility. This paper proposes a comprehensive benchmark for natural language that offers thorough viewpoints on the model’s generalizability and robustness. The benchmark encompasses multiple datasets with varying difficulty levels and characteristics, including temporal shifts and diverse domains. On top of this testbed, we demonstrate that UniDA methods originally designed for image input can be effectively transferred to the natural language domain while also underscoring the effect of adaptation difficulty in determining the model’s performance.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "e8eo25efjm", "reviewer": "Reviewer_17io", "summary": "", "questions": "A. Can you justify why the H-scores of more challenging tasks, such as MASSIVE, are higher than easier tasks?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We’d like to express our gratitude for your insightful and positive comments. Here are the responses to the comments.\n\n\n**W1. Limited Model Size**\n\nAs recent LLMs are capable of a wide variety of real-world applications, we strongly agree on the importance of exploring the LLM’s robustness from a Universal Domain Adaptation (UniDA) perspective.\nHowever, in this current work, we have limited the scope to introducing UniDA by proposing a solid benchmark and validating existing state-of-the-art methods within the existing computer vision and NLP literature.\nTo ensure the closest alignment with the existing literature in NLP, we conducted all the experiments with the same backbone as the previous domain adaptation studies.\nAs the research is still in its early stages, further exploration considering LLMs is a great topic for future work.\n\n\n**W2. Focused on Classification Tasks**\n\nWe acknowledge that our work is limited to text classification tasks, and we fully agree that our work can be expanded to diverse tasks such as NER, POS tagging, and even text generation.\nHowever, the scope of our work was primarily influenced by existing UniDA literature in the vision domain, which is predominantly focused on the classification task.\nAs we are the first to introduce the UniDA concept to the NLP field, we considered it appropriate to begin from the same scope as existing works.\nAdditionally, the selective task preference is also due to the challenge of defining concepts such as domain and category shifts in tasks such as text generation.\nDespite the weaknesses, we believe our work provides sufficient empirical analysis within the text classification tasks and offers a substantial array of insights.\n\n\n**W3. Lack of Zero-shot and Few-shot Models**\n\nAs the majority of models capable of zero-shot and few-shot inference are LLMs, our response remains consistent with W1.\n\n\n**Q1 : Why do some challenging tasks show a higher H-score compared to easier tasks?**\n\nWe proposed two different metrics (PDR, DDS) to capture the challenges of UniDA (domain gap and category gap).\nSpecifically, PDR quantifies the relative performance degradation, irrespective of the absolute difficulty of the task.\nLet us compare the results of MASSIVE and Huffpost (2016) for a better explanation. (Please refer to Table 1 of the paper and the table below).\n\nFor the **MASSIVE** dataset, 98.01 was observed as the source accuracy, indicating that the in-domain task is considerably easy.\nThe target accuracy is 62.58, yielding a high PDR value of 36.14, as calculated by Equation 1 of the paper.\n\nOn the other hand, the **Huffpost (2016)** dataset is notably difficult, even within the in-domain context.\nSpecifically, with a source accuracy of 82.15 and a target accuracy of 61.61, a relatively lower PDR value of 25.00 was observed.\n\nAs a result, MASSIVE exhibits a higher PDR value compared to Huffpost (2016), indicating a relatively challenging scenario (i.e., a more significant domain gap).\nHowever, if we compare the absolute performance in the target domain, MASSIVE reports a higher performance of 62.58, while Huffpost (2016) shows 61.61.\n\nDespite MASSIVE being a challenging task with a high PDR value, it is possible to observe a higher H-score compared to easier tasks such as Huffpost (2016).\n\n|  | Source Accuracy | Target Accuracy | PDR |\n| --- | --- | --- | --- |\n| Huffpost (2016) | 82.15 | 61.61 | 25.00 |\n| MASSIVE | 98.01 | 62.58 | 36.14 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "**Reasons to Accept:**\n\n1. **Novel Topic Exploration:** The paper is one of the first to explore the applicability of Universal Domain Adaptation (UniDA) in the field of NLP, potentially enhancing the robustness of NLP models.\n\n2. **Comprehensive Benchmark and Testbed:** The provision of a comprehensive benchmark and a robust testbed for evaluating UniDA's performance in NLP is a significant strength of the paper. These resources can accelerate research in this area by providing a well-structured foundation for future work.\n\n3. **Introduction of New Metrics:** The paper presents two new metrics - Performance Drop Rate (PDR) and Distinction Difficulty Score (DDS) - which can be instrumental in assessing domain and category gaps. These metrics add to the paper's novelty and can benefit the wider NLP community.\n\n4. **Empirical Evaluation:** The paper provides a thorough empirical evaluation of existing UniDA methods, extending their use from image to language inputs. It provides valuable insights into how these methods perform in a new context, potentially aiding future research efforts in UniDA for NLP.\n\n5. **Benefit to NLP Community:** The research presented in this paper can contribute to improving the robustness of NLP models against distributional shifts, a significant challenge in deploying machine learning systems in real-world scenarios. This is directly beneficial to the NLP community.\n\nOverall, the paper demonstrates the substantial potential to influence future work in the domain adaptation field of NLP, both by providing a novel framework for understanding the application of UniDA and by offering resources for further exploration and development.", "reasons_to_reject": "It is a good work that I do not have strong reasons to reject it, and several limitations have been clearly stated by the authors for future work. \n\n1. **Limited Model Size Evaluation:** The study only conducted evaluations with models of limited sizes. While the authors noted that the evaluation of large language models (LLMs) is part of their future work, the absence of their performance assessment limits the comprehensive understanding of UniDA methods across varying model sizes in the NLP context.\n\n2. **Focused on Classification Tasks:** The authors explicitly mention that their proposed testbed is limited to text classification tasks. While this focus may have been practical, it also restricts the generalizability of the findings. Given the diverse range of tasks in NLP, such as named entity recognition, part-of-speech tagging, and sentiment analysis, this limitation might reduce the applicability of the study's contributions.\n\n3. **Lack of Evaluation of Zero-shot and Few-shot Models:** The current trend in NLP includes models with zero-shot and few-shot learning capabilities, but these were not evaluated in the study. Including such models might have provided a more comprehensive evaluation of UniDA's performance.\n\nIn conclusion, while the paper provides important contributions, it has several limitations that may affect the comprehensiveness and applicability of its findings. These limitations could be addressed in future work to enhance the paper's impact and relevance to the broader NLP community.", "questions_to_authors": "A. Can you justify why the H-scores of more challenging tasks, such as MASSIVE, are higher than easier tasks?", "paper_topic_and_main_contributions": "**Paper Topic:**\n\nThis paper centers around the topic of Universal Domain Adaptation (UniDA) in the field of Natural Language Processing (NLP). UniDA, which has shown promising results in the computer vision field, aims to enable machine learning systems to effectively apply prior knowledge to unfamiliar domains and robustly respond to anomalous inputs. The paper argues that despite the success of UniDA in computer vision, its application in language inputs remains under-explored. \n\n**Main Contributions:**\n\n1. **Comprehensive Benchmark for NLP:** The paper presents a comprehensive benchmark for evaluating the generalizability and robustness of NLP models. This benchmark covers multiple datasets with different difficulty levels and characteristics, including temporal shifts and diverse domains, offering a thorough viewpoint of a model's capabilities.\n\n2. **New Metrics:** To better understand the challenges posed by UniDA in the NLP context, the authors propose two novel metrics: Performance Drop Rate (PDR) and Distinction Difficulty Score (DDS). These metrics are designed to quantitatively measure the domain gap and category gap.\n\n3. **Evaluation of UniDA Methods:** The paper provides an empirical evaluation of existing UniDA methods, originally designed for image inputs, within the context of natural language. It also compares these methods with state-of-the-art domain adaptation techniques from NLP literature.\n\n4. **Insightful Findings:** The study reveals that UniDA methods, initially conceived for image inputs, can be effectively transferred to the natural language domain. It also highlights the significant role of adaptation difficulty in determining the performance of these methods.\n\n5. **Testbed for Evaluating UniDA:** A significant contribution of the paper is the introduction of a testbed for evaluating UniDA's performance in the NLP field. This testbed, featuring various levels of domain and category gaps across different datasets, serves as a valuable resource for future research in this area.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "QaYzrKvFeM", "reviewer": "Reviewer_pfYB", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We genuinely appreciate your thoughtful feedback. Below is our response to your comments.\n\n\n**W1. Incomprehensive Metric (H-score)**\n\nWe agree that the H-score has its limitations.\nHowever, the H-score has consistently been employed across the Universal Domain Adaptation (UniDA) literature ([1], [2], [3], [4]) and is currently perceived as the best metric for measuring the essence of the UniDA objective, \nwhich is to minimize both the domain and category gap simultaneously.\n\nAs the reviewer clearly pointed out, examining the optimal performance of each method for *one certain gap* can also offer intriguing observations.\nWe believe it could be a factor well worth considering in future UniDA research.\nYet, the UniDA objective is not solely limited to handling a single gap - instead, it seeks to robustly handle the concurrent occurrence of both the domain and category gap.\nFrom this standpoint, the H-score, harmonic mean of *transferable* and *unknown* accuracy, is deemed a valid evaluation metric for UniDA.\n\n\n\n**W2. Limited Baseline Methods**\n\nWe have also considered the scenario the reviewer mentioned during the research process.\nThe rationale behind excluding this scenario from our baselines is the following:\n\n- The samples from the target domain are inherently *“out-of-domain”* (OOD) from the perspective of the source domain.\n- OOD detection methods, which aim to detect all out-of-domain inputs, would most likely classify all the samples from the target domain as OOD.\n- If all samples in the target domain were classified as OOD, the criteria for discerning *transferable* and *unknown* inputs become inherently unclear.\n- Hence, we decided to solely employ baselines that first apply adaptation and distinguish *unknown* samples afterward.\n\nOur work follows the early research in the UniDA literature from the vision domain ([1], [2], [5], [6]), which predominantly considers baselines that prioritize applying domain adaptation (DA).\n\n\n[1] Fu et al., Learning to Detect Open Classes for Universal Domain Adaptation. ECCV 2020\n\n[2] Li et al., Domain consensus clustering for universal domain adaptation. CVPR 2021\n\n[3] Chang et al., Unified Optimal Transport Framework for Universal Domain Adaptation. NeurIPS 2022\n\n[4] Chen et al., Evidential Neighborhood Contrastive Learning for Universal Domain Adaptation. AAAI 2022\n\n[5] You et al., Universal Domain Adaptation. CVPR 2019\n\n[6] Saito et al., Universal Domain Adaptation through Self Supervision. NeurIPS 2020", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) This paper provides a new benchmark for a new challenging task. It provides a couple of datasets and a standard metric for future works.\n2) The benchmark is fully investigated and can cover most scenarios of UniDA.", "reasons_to_reject": "1) The proposed metric for the task (section 4.3) is incomprehensive. The authors claim that UniDA is challenging because it has two gaps, but the proposed H-score cannot reflect how well a model can solve one certain gap. As a result, it is also uncertain whether the claimed two challenges are all difficult for existing works or worth studying both.\n2) Baseline methods are limited. The problem is a combination of DA and OOD detection so an intuitive thought is to do the OOD detection first and process the in-domain samples with a DA model. The baselines in this paper are either single DA models for NLP or UniDA models for computer vision but the combination of OOD detection model for NLP and DA model for NLP is missing.\n\n**Edit after rebuttal**\n\nAll my concerns are being answered. But it seems like the work always follows what CV has done but lacks NLP-specific discussion.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a benchmark for Universal Domain Adaptation (UniDA). When transferring from a source domain to a target domain, UniDA requires a model to deal with transferable samples properly while detecting out-of-domain samples. This problem faces two challenges: domain gap requires models to transfer across domains while category shift requires models to distinguish unknown samples from transferable samples. To evaluate the proposed benchmark, the authors propose two measurements to quantize two major challenges of UniDA. Then, extensive experiments show the performance of existing methods on the new benchmark.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1) Line 215: Format of 'bert-base-uncased'\n\n2) The information in Table 1 and Figure 2 is repetitive. Table 1 can be deleted."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper looks at the task of generalization and  domain adaptation borrowing ideas from previous work (Universal Domain Adaptation) from computer vision, and applying them to NLP datasets. The paper presents benchmark tasks, alternate metrics, and empirically evaluate the ideas of Universal Domain Adaptation on the benchmarks and find them to transfer well.

The reviewers have rated the paper strong on soundness but excitement scores vary. The authors address some of the concerns of reviewers in terms of limited baselines, and In particular reviewers have noted several limitations of the work: namely that it is confined to classification tasks, limited model size/architecture, and is missing some zero-shot baselines from large language models which would have been appropriate for this topic. Nevertheless the experiments are mostly sound even if missing interesting comparisons.
Decision: Accept-Findings