Domain Adaptation for Sentiment Analysis Using Robust Internal Representations
Authors: Mohammad Rostami
Digbalay Bose
Shrikanth Narayanan
Aram Galstyan
Venue: EMNLP 2023 Findings (2023)
Keywords: domain adaptation
sentiment analysis

Abstract: Sentiment analysis is a costly yet necessary task for enterprises to study the opinions of their customers to improve their products   and to determine optimal   marketing strategies.  Due to the existence of a wide range of domains across different products and services, cross-domain sentiment analysis methods have received significant attention. These methods mitigate the domain gap between different applications by training cross-domain generalizable classifiers which relax the need for   data annotation for each domain. We develop a   domain adaptation method which induces large margins between data representations that belong to different classes in an embedding space. This embedding space is trained to be domain-agnostic by matching the data distributions across the domains. Large interclass margins in the source domain help to reduce the effect of ``domain shift''   in the target domain. Theoretical and empirical analysis are provided to demonstrate that the proposed method is effective.

Domain Adaptation for Sentiment Analysis Using Robust Internal Representations
Mohammad Rostami
Digbalay Bose
University of Southern California rostamimeusc. edu
University of Southern California
dboseêusc. edu
Shrikanth Narayanan
Aram Galstyan
University of Southern California shr bhri@usc.edi
University of Southern California galstyan@usc. edu
Abstract
2020; Gong al., 2020; Xi et al., 2020; Dai al.,
2020; Lin et al., 2020; Long al., 2022; Badr al.,
Sentiment analysis is costly yet necessary task for enterprises to study the opinions their customers improve their products and to de- termine optimal marketing strategies. Due to the existence a wide range domains across different products and services, cross-domain sentiment analysis methods have received sig- nificant attention. These methods mitigate the domain gap between different applications by training cross-domain generalizable classifiers which relax the need for data annotation for each domain. We develop domain adaptation method which induces large margins between data representations that belong different classes in an embedding space. This embed- ding space is trained to be domain-agnoste by matching the data distributions across the do- mains. Large interclass margins the source domain help to reduce the effect of "domain shift" in the target domain. Theoretical and empirical analysis are provided to demonstrate that the proposed method is effective.
2022; Ryu et al., 2022; Huang et al., 2023). The goal is to relax the need for annotation via transfer- ring knowledge transfer from another domain with annotated data to train models for domains with unannotated data.
The above problem has been studied more broadly in the domain adaptation (DA) litera- ture (Wang and Deng, 2018). common DA ap- proach align the distributions of both domains in a shared embedding space (Redko and Sebban, 2017). As a result, a source-trained classifier that receives its input from the embedding space will generalize well in the target domain. In the senti- ment analysis problem, this means that polarity of natural language can be expressed independent of the domain in the embedding space transcendent discrepancies. We can model the embedding space using a shared deep encoder trained to align the distributions at its output space. This training pro- cedure has been implemented indirectly using ad- versarial learning (Li et al., 2019; Dai et al., 2020; El Mekki et al., 2021; Jian and Rostami, 2023) by directly minimizing loss functions that are de- signed align two probability distributions (Kang et al., 2019; Guo et al., 2020; Xi al., 2020; Lin et al., 2020; Rostami and Galstyan, 2023a).
1 Introduction
The main goal in sentiment classification is to pre- dict the polarity of users automatically after collect- ing their feedback using online shopping platforms, e.g., Amazon customer reviews. A major challenge for automatic sentiment analysis is that polarity is expressed using completely dissimilar terms and phrases in different domains. For example, while terms such as "fascinating" and "boring" are used to describe books, terms such as "tasty" and "stale" are used to describe food products. As a result of this discrepancy, an NLP model that is trained for particular domain may not generalize well in other different domains, referred as the problem of "domain gap" (Wei al., 2018). Since generating annotated training data for all domains is expen- sive and time-consuming (Rostami al., 2018), cross-domain sentiment analysis has gained signifi- cant attention recently (Guo et al,, 2020; Du et al.,
Contributions: Our idea based on learning parametric distribution for the source domain in cross-domain embedding space (Rostami and Galstyan, 2023b). We estimate this distribution Gaussian mixture model (GMM). We then the GMM distribution align the source and the target distributions using confident samples, drawn for them GMM increase the interclass margins to reduce the domain gap.
2 Related Work
While domain adaptation methods for visual do- mains usually use generative adversarial networks

---

(GANs) (Goodfellow et al., 2014),the dominant approach for cross-domain sentiment analysis is to design appropriate loss functions that directly impose domain alignment. The main reason is that natural language is expressed in terms of dis- crete values such as words, phrases, and sentences. Since this domain is not continuous, even if we convert natural language into real-valued vectors, it will not be differentiable. Hence, adversarial learn- ing procedure cannot be easily adopted for natural language processing (NLP) applications. The al- ternative approach is to minimize a probability dis- tribution metric to reduce domain cap (Shen et al., 2018). For example, using Wasserstein distance (WD) for domain alignment in visual domains has been found to be highly effective (Long et al., 2015; Sun and Saenko, 2016; Lee al., 2019; Rostami et al., 2019). We rely on the sliced Wasserstein dis- tance (SWD) for aligning distribution because of having less computational load (Lee et al., 2019).
target domain T with an unannotated dataset DT = (Xs), where XT i.i.....,.7 exyl E X C RdxM, The real-valued feature vectors Xs and X are obtained after pre-processing the input text data using common NLP methods, e.g., bag of words or word2vec. We consider that both domains share the same type of sentiments and hence the one-hot labels y encode k sentiment types, e.g.. negative or positive in binary sentiment analysis. We also assume that the source and the target fea- ture data points are drawn independently and iden- tically distributed from the domain -omaiin-peccfic distri- butions z5 ~ ps(x)) and ø. ~ pr(x), such that (r(x) A ps(()), i.e., there exists domain gap.
Given family of parametric functions fo Rd -> V. e.g., deep neural networks with learnable weights 0, and considering an ideal labeling function f(-), e.g., V(x, y) : y = f(x), the goal is to search for the optimal pre- dictor (-)(-) in this family for the target domain. This model should have minimal expected error, i.e., H" arg mino arg ming (Er) (-) is a proper loss function and B(-) denotes the expectation operator. Since the target domain data is unlabeled, the naive approach is to estimate the optimal model using the standard empirical risk minimization (ERM) in the source domain:
The major reason for performance degradation of source-trained model in target domain stems from "domain shift", i.e., the boundaries between the classes change in the embedding space which in turn increases possibility of misclassification. has been shown that if an increased marginn clas- sifier is trained in the source domain, it can gen- eralize better than many methods that try to only align distributions (Tommasi and Caputo, 2013; Rostami, 2022). Inspired by this argument, our method is based on both aligning distributions in the embedding space and also inducing larger mar- gins between classes by learning "parametric dis- tribution" for the source domain. Our idea is based on the empirical observation that when a deep net- work classifier is trained in a domain with anno- tated data, data points of classes form separable clusters in the embedding space modeled via the network responses in hidden layers. This means that the source distribution can be modeled multimodal distribution in the embedding space. Our work is based on using the multimodal distri- bution to induce larger margins between the class- specific clusters after an initial training phase in the source domain.
fo(at))), where
i = arg min (Co(Xs, Ys. C)}
= aar min 1
Σ ((f((a))))))+
Given a large enough labeled dataset in the source domain, ERM model generalizes well in the source domain. The source-trained model may also per- form better than chance in a similar target domain, given cross-domain knowledge transfer, yet its per- formance will degrade in the target domain com- pared to its performance in the source domain because of existing distributional discrepancy be- tween the two domains, ps + PT. Our goal is benefit from the encoded information in the un- labeled target domain data points and adapt the source-trained classifier fg to generalize better in the target domain. We use the common approach of reducing the domain gap across the two domains by mapping data points into a shared embedding. We consider that the predictor model fa(-) can be decomposed into a deep encoder subnetwork @v(+) :X))x)" + Z C Rp and a classifier subnetwork hyp ()) : Z + y such that fo = hag o Ou. where
3 Cross-Domain Sentiment Analysis
Consider two sentiment analysis problems in a source domain with an annotated dataset Ds = (Xs, Ys), where Xs = [zj..... xy1 E X C N and Ys = [er, vyl E y R and

---

Source Domain data
Sowrce Domnin Data
Embedding Space
Embedding Space
Encoding
Encoding
Larger
Telostuttie. Mhuch..
Enconler
Classtteee
sdkmioutee Workk, Woeth
Encwder
Classiter
Probetopical Sameples
1. The model pre-trained the saurce domain
2. GMM is estimated in the embedding space
3. The model adapted on the target pergetidon domain using both CMM samples and source data
Target Domain data
Embedding Space
Embedding Space
Domain Shift effect is miliggated Larger Margins
Domain Shift Caused by Domain Gap
Post-Adaptation
O Improved Generalization
Target Domain Data
Poor Generalization
Due to Small Margins
Figure I: Architecture of the proposed cross cross-domin domain sentiment analysis framework. Left: separable data clusters are formed in the embedding space after initial supervised model training in the source domain and then following our previous work (Rostami, 2021), the internally learned multimodal distribution is estimated using GMM. Right: random samples from the GMM with high-confiden labels drawn and then used generate pseudo-datase The pseudo-dataset helps to induce larger margins between the classes to mitigate domain shift the target domain.
8 8= (w, v). Here, Z is embedding space which is modeled by the encoder responses its output. We assume that the classes have become separable for the source domain this space after an initial training phase on the source domain (see Figure 1, left). If we can adapt the source-trained encoder network such that the two domains share similar distributions 2, i.e., (-)s) x (-)r)(),, the em- bedding space would become domain- agnostic. As result, the source- traiced classifier network will generalize well in the target domain. A number of prior cross-domain sentiment analysis algorithms use this strategy, select proper distribution metric to compute the distance between ((ps(x") and Φ(P (x²)), and then train the encoder network to align the domains via minimizing it:
puted using the empirical samples that drawn from the two distributions. Finally, SWD possesses non-vanishing gradient even when the support of the two distributions do not overlap (Bonnotte, 2013; Lee et al., 2019). Hence SWD is suitable for solving deep learning problems which are normally handled using first-order gradient-based optimiza- tion techniques, e.g., Adam.For more background SWD please refer to the Appendix (Section B). While methods based on variations of Eq. (2) are effective to reduce the domain gap some extent, our goal to improve upon the baseline obtained by Eq. (2) by introducing a loss term that increases the margins between classes in the source domain (check the embedding space in Figure 1, right, for having better intuition). By doing so, goal is to mitigate the negative effect of domain shift.
min
w,e0 N Σ ((hm(o
Increasing Interclass Margins
+ (0( (...).....rr. (pr(())))
Our idea for increasing margins between the classes is based on learning an intermediate parametric distribution in the embedding space. We demon- strate that this distribution can be used to induce larger margins between the classes. To this end, consider that the elassifier subnetwork consists of a softmax layer. This means that the classifier should become maximum a posteriori (MAP) estimator after training to able to assign a mem- bership probability to a given input feature vec-
where .).,)) denotes a probability metric to mea- sure the domain discrepancy and À is a trade-off parameter between the source ERM and the do- main alignment term. We base our work this general approach and use SWD (Lee al., 2019) compute ----)) in (2). Using SWD has three ad- vantages. First, SWD can be computed efficiently compared to WD based on a closed form solution WD distance in 2D. Second, SWD can be com-

---

Algorithm SAIM (A,
for estimating the GMM parameters, the source domain data points are labeled. As result, we can estimate μi and En for each component inde- pendently using standard MAP estimates. Simi- larly, the weights 0, can be computed by a MAP estimate. Let Sj denote the support set for class in the training dataset, i.e., Sj = {( yf) e Ds mrx j}. To cancel out outliers, we include only those source samples in the S, sets, for which the trurne-trained model predicts the cor- responding labels correctly. The closed-form MAP estimate for the mode parameters is given as:
I: Initial Training:
2: Input: source dataset Ds = (Xs, Ys),
3: Training on the Source Domain:
4: do (wo, éb)
5: arg mins (if(So(a)), vf)
6: Prototypical Distribution Estimation:
7: Use (4) and estimate Oij. MMj Ej
8: Model Adaptation:
9: Input: target dataset Dj= = Xy
10: Pseudo-Datasel Generation:
11: Dp= (Zp, Yp)-
(I=I..... N 1 ................................ I).). where:
12:
pz(;),
Np
Syl
Σ
|Sy] (o()),
y arg j inio (z.)) =)
15:
max hamu (z > T
Σ
Σ (o (ES)
[, (0.((x)) - Ais) (d)(aa)) - -Az).
16:
for itr = 1,.... ITR do
17:
draw data batches from Ds. Dy, and Dp
(4)
18:
Update the model by solving (6)
Computations in Eq. (4) can be done efficiently. For a complexity analysis, please refer to the Ap- pendix (Section C). Our idea is to use this mul- timodal GMM distributional estimate to induce larger margins in the source domain (see Figure I, right). We update the domain alignment term in (2) induce larger margins. To this end, aug- ment the source domain samples in the domain alignment term with samples of labeled pseudo- dataset Dp = (Zp, Yp) that we generate using the GMM estimate, where Zp = [z!...., ....,, NN Np Yp [y'...., "ff........ -%,,]
19:
end for
tor. Under this formulation, the model will gen- eralize in the source domain if after supervised training of the model using the source data, the in- put distribution transformed into multi-modal distribution (-)-) (-)) with & modes in the embedding space (see Figure 1, left). Each mode of this distribution corresponds to one type of sentiments. The geometric distance between the modes of this distribution corresponds to the margins between classes. If we test the source- trained model in the target domain, the boundaries between class modes will change due "domain shift", i.e., de (-) + (-)) As visualized in Figure 1, if we increase the margins between the speciife modes in the source domain, domain shift will likely cause less performance degrada- tion (Tommasi and Caputo. 2013).
R This pscudo-dataset is generated using the the GMM distribution. We draw samples from the GMM distributional estimate z! ~ Ps(²) for this pur- pose. To induce larger margins between classes, feed the initial drawn samples into the classi- fier network and check the confidence level of the classifier about its predictions for these randomly drawn samples. We a confidence threshold level т and only select a subset of samples for which the confidence level of the classifier is more than T:
We estimate the multimodal distribution in the embedding as a Gaussian Mixture Model (GMM):
ps(z) ay (zlu,, E,),
(ef (.) Dp if: 2. ~ p)(()) and max max(M(z{) > vf = arg max(((x))))
(5)
where μ1 and Σj denote the mean and co- variance matrices for each component and ay denotes mix- ture weights for each component. We solve for these parameters estimate the multimodal dis- tribution. Note that unlike usual cases in which iterative and tmne-cnnu algorithms such as ex- pectation maximization algorithm need to be used
Given the GMM distributional form, selection of samples based on the threshold means that we include GMM samples that are closer to the class- specific mode means 14 (see Figure 1). The mar- gins between the clusters the source domain increase if we use the generated pseudo-dataset for

---

domain alignment. Hence, we update Eq. (2) as:
for the target expected error:
er Ses + Díus, Ar) + D(ūr, jov) + (1- -)) EsS
1 Σ ((hn(
o, arg min
N
/e2loa 1 1/9 N + M +2
1
Np N Σ ((ww(()) v.)
(6)
No
where ç is constant depending C(-).
+ (Xs) (x)):
The complete proof is included in the Appendix (Section C).
Proof:
The first and the second terms in (6) are ERM terms for the source dataset and the generated pscudo- dataset guarantee that the classifier continues to generalize well in the source domain after adapta- tion. The third and the fourth terms are empirical SWD losses (sce Appendix for more details) that align the source and the target domain distributions using the pseudo-dataset which as we describe in- duces larger margins. The hope is that as visualized in Figure 1, these terms can reduce the effect of domain shift. Our proposed solution, named Sen- timent Analysis using Increased-Margin Models (SAIM²), is presented in Algorithm 1 and Figure 1,
Theorem provides an explanation to justify Algorithm 1. We observe that all the terms in the upperbound of the target expected error in the right- hand side of (7) are minimized by SAIM² The source expected error is minimized as the first term in (6). The second and the third terms are mini- mized the third and fourth terms of (6). The fourth term 1 T will be small if we set T 1. The term 6s.7 is minimized through the first and the second term of (6). This is highly important as using the pseudo-datase provides a way to min- imize this term. As can be seen in our proof in the Appendix, if we don't tt use the pseudo-dataset, this terms is replaced with eS,T which cannot be minimized directly due to lack of having annotated data in the target domain. The last term in (7) is constant term that as common in PAC-learning can become negligible states that in order to train a good model we have access to large enough datasets. Hence all the terms in the upperbound are minimized and if this upperbound is tight, then the process leads to training a good model for the target domain. If the two domain are related, e.g., share the same classes, and also classes become separable in the embedding space, i.e.. GMM esti- mation error for the source domain distribution in
5 Theoretical Analysis
Following a standard PAC-learning frame- work (Shalev-Shwartz and Ben-David 2014), we prove that Algorithm 1 minimizes an up- perbound for the target domain expected eiror. Consider that the PAC-learning hypothesis class to be the family of classifier sub-networks H (-m(())) Z -> R* (R)), where V denotes the number of learnable parameters. We represent the expected error for a model (-) E H. on the source and the target do- mains by (w) and e7(w), Given the source and the target datasets, we can represent the empirical source and target distributions in the embedding space jAs + E ((0v(x;)) and Íoy = 37 Em= (x=()) Similarly, we can build an empirical distribution for the multimodal distribution üP ,, E ((29). In our analysis we also use the notion of joint- optimal model h.,,(()) which w" = arg min CS,T = arg min (cs + eT) for any given domains S and T. When we have labeled data in both domains, this is the best ERM-trained model. Existence of good joint-trained model guarantces that the domains related for positive transfer, e.g., similar sentiment polarities are encoded across the two domains.
the embedding space is small, then the upperbound is going to be likely tight. However, we highlight that the prospect of a tight upperbound is a condi- tion for our algorithm to work in practical settings. Note, however, this is common limitation for most parametric machine learning algorithms.
defined as:
6 Experimental Validation
We have selected the most common setup for UDA sentiment analysis to perform our experiments for possibility of comparing our performance against prior works. Our implemented code publicly available: https//githuh.comddigosz222
6.1 Experimental Setup
Theorem 1: Consider that we use the procedure senti-
described in Algorithm for ment analysis, then the following inequality holds
Dataset and the cross-domain tasks: Most exist- ing works in cross domain sentiment analysis re-

---

port performance on cross-domain tasks that are de- fined using the real-world Amazon Reviews bench- mark dataset (Blitzer et al., 2007). The dataset is built using Amazon product reviews from four product domains: Books (B), DVD (D), Electronics (E), and Kitchen (K) appliances. Hence, 12 pair- wise cross-domain tasks can be defined amongst the domains. Each review is considered to have positive (higher than 3 stars) negative (3 stars or lower) sentiment. Each task consists of 2000 labeled reviews for the source domain and then 2000 unlabeled reviews for the target domain, and 2500-5500 examples for testing. We report our per- formance on the 12 definable cross-domain tasks. We report the average prediction accuracy and stan- dard deviation (std) over 5 runs of our code.
DSN and CMD are similar SAIM in that both align distributions an embedding space. DSN learns shared and domain-specific knowledge for each domain and aligns the shared knowledge us- ing the mean-based maximum mean discrepancy metric. CMD uses the central moment discrepancy metric for domain alignment, ASYM benefits from the idea of pseudo-labeling of the target samples to updated the base model. MT-Tri is based on ASYM but it also benefits from multi-task learning. TRL and PBLM do not use distribution alignment and are based on the pivot based language model. TAT is a recent work that has used adversarial learning successfully for cross cross-domain sentiment analysis. All the methods except TAT that uses 30000 di- mensional features, use 5000 dimensional features. Note that in the results, the methods are comparable if they use features with the same dimension for fair comparison. We report performance of the source only (SO) model as lowerbound to demonstrate the effect of adapting the model over this base- line. We also compare against 4 BERT-based meth- ods: vanilla fine-tuned BERT (VBERT), HATN (Li et al., 2018) which is based on a hierarchical at- tention transfer mechanism, adversarial training over VBER (AT-BERT), and BERT-DA (Du al., 2020). We provided results by the authors in our table. We report std if std is reported in the original paper.
Preprocessing: We have used two methods for backbone feature extraction: if-idf and BERT fea- tures (Du et al., 2020). The most common setup in the literature is using If-idf feature vector of bag-of-worte words unigrams and bigrams, where we encode each review as a d 5000 dimensional or d = 30000 dimensional if-idf feature vectors. Following (Du et al., 2020), qe also report perfor- mance when modern BERT features are used. Fol- lowing the precedence, we first fine-tune BERT backbone, followed by FCC layers, on the source domain solve the sentiment analysis task. We call this baseline vanilla BERT (VBERT), ana- logues to SO for if-idf features. Features extracted via VBERT are then used by the BERT-based meth- ods for domain adaptation. Note that using BERT as an advanced text embedding method increases the absolute performance, i.e., pretraining of BERT on large auxiliary dataset enables knowledge transfer. However, in model udaptation we must evaluate the algorithms according to the relative improvement of the model performance on the tar- get domain compared to the solely source-trained model. Hence, only relative improvement over the baseline should be used for evaluating performance of domain adaptation algorithms.
Model and optimization setup: We used the Ceature-specifir benchmark neural network archi- tecture that is used in the above mentioned works for fair comparison. For tf-idf-based comparison, we used an encoder with one hidden dense layer with 50 nodes with sigmoid activation function. The classifiers consist of a softmax layer with two output nodes. We implemented our method in Keras, used adam optimizer and tuned the learning rate in the source domain. For BERT-based archi- tecture, we have concatenated the BERT backbone with two fully connected layers with 256 and 2 nodes and ReLU and SoftMax nonlinear functions. Hyperparameter tuning: An advantage for our algorithm is that there are only two major hyperpa- rameters, and X. We set 0.99 and À = 10 We tuned À based a brute-force search. Note, however, we observed empirically that our algo- rithm is not sensitive the value of A. We used a GPU cluster equipped with Nvidia Tesla P100- SXM2 GPUs. We used Keras for implementation!
We compare our method against several existing algorithms that use if-idf features. We compare against DSN (Bous- malis et al., 2016) CMD (Zellinger et al., 2017), ASYM (Saito et al., 2018), PBLM (Ziser and Re- ichart, 2018), MT-Tri (Ruder and Plank, 2018). TRL. (Ziser and Reichart, 2019), and TAT (Liu et al., 2019). These works are representative of advances in the field based on various approaches.
Methods for Comparison:
The code is available as supplement.

---

Features
Method
B+D
B-E
B-K
D-B
D-EE
D-*K
SO
81.7 + 0.2
74.0 + 0.6
76.4 = 1.0
74.5 0.3
75.6 - 0.7
79.5 - 0.4
DSN
82.8 0.4
81.9 + 0.5
84.4 0.6
80.1 + 1.3
81.4 1.1
83.3 + 0.7
CMD
82.6 0.3
81.5
0.6
4.3
80.7 + 0.6
82.2 0.5
84.8 0.2
ifiidf
ASYM
80.7
79.8
82.5
73.2
77.0
82.5
d 5000
PBLM
84.2
77.6
82.5
82.5
79.6
83.2
MT-Tri
81.2
78.0
78.8
77.1
81.0
79.5
TRL
82.2
82.7
SATM
83.2 + 0.2 83.9 + 0.3
85.9 + 0.3
80.3 + 0.4
84.2 + 0.3 87.3 0.2
if-idf
TAT
84.5
80.1
83.6
81.9
81.9
84.0
d = 30000
SAIM²
86.2 + 0.2 85.1 + 0.2 87.6 + 0.2 80.9 + 0.5
85.2 + 0.2 88.6 = 0.2
VBERT
89.0
88.0
89.1
89.4
86.6
87.5
VBERT
85.3
85.8
88.8
85.0
85.8
87.0
HATN
89.4
87.2
89.4
89.8
87.0
87.6
BERT
AT-BERT
89.7
87.3
89.6
89.6
86.1
87.7
BERT-DA
89.8
88.1
90.7
90.4
88.1
88.6
SAIM2
87.5
88.3
88.0
90.5
87.3
88.5
Features
Method
E-+B
E-D
E-+K
K-+B
K-+
K-+E
SO
72.3 + 74.2
0.6
85.6 + 0.6
73.1
0.1
75.2
0.7
85.4 +
DSN
75.1 + 0.4
77.1 + 0.3
87.2 + 0.7
76.4
0.5
78.0 + 1.4
86.7 = 0.7
CMD
74.9 + 0.6
77.4 + 0.3
86.4 + 0.9
75.8 + 0.3
77.7 + 0.4
86.7 = 0.6
id
ASYM
73.2
72.9
86.9
72.5
74.9
84.6
d = 5000
PBLM
71.4
75.0
87.8
74.2
79.8
87.1
MT-Tri
73.5
75.4
87.2
73.8
77.8
86.0
TRL
75.8
72.1
S2IM
78.6 + 0.4 79.7 + 0.2 89.2 0.2 76.7 + 0.4
79.1 + 0.4
87.0 + 0.1
if-idf
TAT
83.2
77.9
90.0
75.8
77.7
88.2
d = 3000
SATM2
78.8 + 0.3 78.9 + 0.3 90.1
0.2
78.1 + 0.2 78.8 + 0.4 88.1 + 0.1
VBERT
86.5
86.2
91.6
87.6
87.3
90.5
VBERT
84.3
78.8
86.0
84.5
81.0
87.0
HATN
87.2
88.8
92.0
87.9
87.9
90.3
BERT
AT-BERT
87.2
88.2
91.9
87.7
87.7
90.3
BERT-DA
88.3
89.0
92.8
87.9
88.4
90.6
SAIM2
89.0
85.5
90.8
88.0
84.5
91.3
Table I:
Classification accuracy for the cross-domain sentiment analysis tasks for the Amazon Reviews dataset. Bold font denotes the method with maximum performance in each column for the corresponding inOut feature.
6.2 Comparative Results
more similar our algorithm- have relatively sim- ilar performances. This observation suggests that we should not expect considerable performance boost if we simply align the distributions by de- signing a new alignment loss function. This means that outperformance of SAI M2 compared to these methods stems from inducing larger margins. We verify this intuition in our ablative study. We also observe that increasing the dimension of if-idf fea- tures 30000 leads to performance boost which is probably the reason behind good performance of TAT. We conclude that for a fair comparison on
Our results are reported in Table 1. In this table, bold font denotes the best performance among the methods that use the same feature type. For the case of tf idf with - 5000 which is the classic set- ting, we see that SATM2 algorithm performs rea- sonably well and in most cases leads to the best per- formance. Note that this is not unexpected as none of the methods has the best performance across all tasks. We observe from this table that overall the methods DSN and CMD which are based on align- ing the source and target distributions- which are

---

these tasks, we should always use the same dimen- sion generate the features.
For the BERT-based methods, have reported two versions of vanilla BERT. VBERT denotes re- sults reported by (Du et al., 2020) and VBERT denotes the results that we could obtain following all the guidelines that are provided in (Du et al., 2020). We first note that vanilla BERT performance are considerably higher than SO formance. This significant boost is likely due the pretraining of BERT and should not be considered a reason for improved cross-domain performance. We also note that our performance competitive for many tasks. We note that lesser performance of our method some cases stems from smaller VBERT* performance versus VBERT. We speculate our VBERT perfor- mance can be boosted to performance similar to VBERT* through better optimization hyperparame- tuning. However, if base our comparison on the relative improved performance that is gained as the result of the adaptation procedure, our model highly competitive. As also predictable from our theoretical result, we empirically conclude that the initial source-only performance of the model on the target domain is extremely important.
(a) Source
(b) GMM. Dist. Samples
per-
external large corpus
(c) Target: Pre-Adapt.
(d) Target: Adst-A
Figure UMAP visualization for the D-*K task: (a) the source domain testing split, (b) the GMM distribu- tion samples, (c) the target domain testing split pre- and (d) post-adaptation (Best viewed in color).
83.0
Accuracy
87.6
To provide an empirical exploration to validate the intuition we used for our rationale, we have used UMAP (MeInnes et al., 2018) to reduce the dimension of the data representations in the 50D embedding space two for the purpose of 2D vi- sualization. Figure 2 visualizes the testing splits the source domain before model adaptation, the testing splits of the target domain before and af- ter model adaptation, and finally random samples drawn from the estimated GMM distribution for the D-*K task. Each point in the figure represents data point and each color represents one of the sentiments. Observing Figure 2a and Figure 2b, we conclude that the estimated GMM distribution approximates the source domain distribution rea- sonably well and at the same time, a margin be- tween the classes in the boundary region is ob- servable. Figure 2c visualizes the target domain samples prior to model adaptation. As expected, observe that domain gap has caused less sep- arations between the clusters for the classes, as also evident from SO performance in Table 1. Fig- ure 2d visualizes the target domain samples after aduptation using SA/M² algorithm. Comparing Figure 2d with Figure 2c and Figure 2a, we see
62.4
0.0
0.2
0.4
0.6
0.6
1.0
Tau
Figure 3: Effect r on performance.
that the classes have become more separated. Also, careful comparison of Figure 2d and Figure 2b veals S.A M2 algorithm has led bias in the target domain move the data points further away from the boundary. These visualizations serve as an empirical verification for our theoretical analysis.
6.3 Ablation Studies
First note that the source only (SO) model and vanilla BERT results, which are trained using (1), already serve as a basic ablative experiment to study the effect of domain alignment. Observed im- provements over this baseline demonstrate positive effeet of domain adaptation performance. The amount of relative improvement also measures the quality of model adaptation algorithms. Addition- ally, we provide two sets of ablative experiments. In Table 2, we have provided an additional abla-

---

Table 2: Performance for AO.
tributions in shared embedding space and induc- ing larger margins between the classes using an intermediate multi-modal GMM distribution. We theoretically demonstrated that our approach min- imizes an upperbound for the target domain error. Our experiments demonstrate that our algorithm is effective and compares favorably against the state- of-the-art. A future research direction is to address cross-domain sentiment analysis when distributed source domains exist (Stan and Rostami, 2022).
Task
B->D
B-+E
B K
81.9 + 0.5
80.9 + 0.8
83.2 + 0.8
Task
+B
D->E
D >
74.0 +: 0.9
80.9 + 0.7
83.4 + 0.6
Task
E E->B
E->D
E->K
74.1 + 0.6
74.0 + 0.3
87.8 + 0.9
Task
D +B
D >
D >
74.1 + 0.6
74.0 0.3
87.8 = 0.9
Limitations
tive experiment to study the effect of the increased interclass margin performance. We have re- ported result of alignment only (AO) model adap- tation based on Eq. (2). The AO model does not benefit from the margins that SAIM² algorithm induces between the classes via solving Eq. (6).
Our approach is based on aligning the distribution by assuming that initially the two distributions have nontaxable overlap. However, if the domain gap is initially large, our approach may fail to align the distributions class-conditionally. As result, some classes might be aligned incorrectly, despite the fact that the two domains end up having similar distributions. To resolve this limitations, we need to use techniques such as pseudo-labelingg methods. Furthermore, our approach operates under the assumption that the source domain data is read- ily accessible and can be directly utilized during the adaptation phase. However, there are situa- tions where the source data private and cannot be shared. In order to address this issue, it becomes crucial for us to devise a mechanism that can ef- fectively align the distributions of the source and target domains without relying on direct usage of the source domain data. To tackle this challenge, need to explore alternative methods that indirectly capture the relevant information from the source domain without compromising its privacy. This requires developing innovative techniques that leverage shared latent representations or intermedi- ate feature spaces for domain alignment. By estab- lishing a connection between the source and target domains at higher level, we may be able to align their distributions and bridge the gap between them without directly sharing data.
Comparing AO results with Table 1, we can con- clude that the effect of increased interclass mar- gins is important in our final performance. We can clearly see that compared to other cross-domain sentiment analysis methods, the performance boost for our algorithm stems from inducing large mar- gins. This suggests that researchers may need to investigate secondary mechanism for improving domain adaptation in NLP domains, in addition to probability distribution alignment.
We have also studied the effect of the value of the confidence parameter on performance. In Figure 3, we have visualized the performance of our algo- rithm for the task B D when is varied in the interval [0, 0.99), where the solid black line shows average performances and the shaded blue region shows the standard deviation. When т = 0, the samples are not necessarily confident samples. We observe that as we increase the value of 7 towards 1, the performance increases as a result of inducing larger interclass margins. For values > 0.8, the performance has less variance which suggests ro- bustness of performance if 1. These empirical observations about 7 values validate our theoretical result as stated in the upperbound (7).
Finally, as large language models (LLMs) are being increasingly adopted in many applications, the effect of domain gap can be mitigated because LLMs have been pre-trained on extremely large corpus that contain large amount of information. As a result, they are applicable to many domains without being affected by domain shift. As resuit, UDA may not be as significant challge as it used to be, but still if domain shift exists a target domain, using UDA is going to be very beneficial.
Finally, our empirical exploration demonstrates that our algorithm robust with respect to data im- balance which is an important advantage in domain adaptation application. Due to the space limit, this study is included in the Appendix (Section E).
7 Conclusions
We developed method for cross cross-domann domain sentiment analysis based on aligning two domain-specinn dis-

---

References
Guoliang Kang. Jiang. Yi Yang. and Alexander G Haupt- mann. 2019. Contrastive adaptation network unsuper- vised domain adaptation. Proceedings of CVPR, pages 4893-4902.
Haidi Badr, Nayer Wanas, Magda Fayek. 2022. Unsu- pervised domain adaptation with post-adaptation labeled domain performance preservation. Machine Learning with Applications, 10: 10:100439.
Chen- Yu Lee, Tammay Batra, Mohammad Haris Baig, and Daniel Ulbricht. 2019. Sliced wasserstein discrepancy for unsupervised domain adaptation. In Proceedings of the CVPR, pages 10285-10295.
John Blitzer, Mark Dredze, and Fernando Pereira. 2007. Bi- ographies, bollywood, boom- boxes and blenders: Domain adaptation for sentiment classification, In Proceedings of ACL, pages 440-447.
Zheng Li, Xin Li, Ying Wei, Lidong Bing, Yu Zhang, and Qiang Yang. 2019. Transferable end-to-end aspect- based sentiment analysis with selective adversarial learning. In Proceedings of EMNLP- pages 4582-4592.
François Bolley, Amaud Guillin, and Cédric Villani. 2007. Quantitative concentration inequalities for empirical mea- sures on non-compact spaces. Probubility Theory and Related Fields, 137(349):51--533
Zheng Li. Ying Wei, Yu Zhang, and Qiang Yang. 2018. Hier- archical attention transfer network for cross-domain senti- classification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32.
Bounotte. 2013. Unidimensional and evolution methods for optimal transportation. Ph.D. thesis, Paris 11.
Konstantinos Bousmalis, George Trigeorgis, Nathan Silber- man, Dilip Krishnan, and Dumitru Erhan. 2016. Domain separation networks. In Proceedings of NewrlPS, pages 343-351.
Chuang Lin, Sicheng Zhao, Lei Meng, and Tat-Seng Chua. 2020. Multi-source domain adaptation for visual sentiment classification. In AAAI, pages 2661-2668.
Yong Dai, Jian Liu, Xiancong Ren, and Zenglin Xu. 2020. Ad- versarial training based source unsupervised domain adaptation for sentiment analysis. In AAAI.
Hong Liu, Mingsheng Long, Jianmin Wang, and Michael Jordan. 2019. Transferable adversarial training: A general approach to adapting deep classifiers. In ICML, pages 4013 4013-4022.
Chunning Du, Huifeng Sun, Jingyu Wang. Qi Qi, and Jianxin Liao. 2020. Adversarial and domain- aware bert for cross- domain sentiment analysis. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 4019 9028..
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jor- dan. 2015. I Learning transferable features with deep adap- tation networks. In Proceedings of ICML, pages 97 105.
Abdellah EI Mckki, Aldelkader El Mahdaouy, Ismail Berrada, and Ahmed Khoumsi. 2021. Domain adaptation for arabic cross-domann and cross-dialect sentiment analysis from contextualized word embedding. In Proceedings of the 2021 Conference of the North American Chapter of the As- sociation for Computational Linguisticx: Human Language Technologies, pages 2824-2837.
Quanyu Long, Tianze Luo, Wenya Wang, and Sinno Pan. 2022. Domain confused contrastive learning for unsupervised do- main adaptation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies. pages 2982-2995.
Leland Mclnnes, John Healy. Nathaniel Saul, and Lukas GroBherger. 2018 UMAP: Uniform manifold approx- imation and projection. Journal of Open Source Soft., 3(29):861.
Chenggong Gong, Jianfei Yu, and Rui Xia. 2020. Unified feature and instance based domain adaptation for end-to- end aspect-based sentiment analysis. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7035-7045.
Todd K Moon. 1996. The expectation maximization algo- rithm. IEEE Signal processing magazine, 13(6):47-00.
Ian Goodfellow, Jean Pouget-Abadic. Mehdi Mirza, Bing Xu. David Wande-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In
R. Neal. 2003. Slice sampling. Annals of statistics, pages 705-741.
Advances in Neural Info. Proc. Systems, pages 2672-2680.
A. Redko. Land Habrard and M. Sebban. 2017. Theoret- ical analysis of domain adaptation with optimal trans- port. In Joint European Conference on Machine Learning and Knowledge Discovery in Dotabases, pages 737-753. Springer.
Han Guo, Ramakanth Pasunuru, and Mohit Bansal. 2020. Multi-source domain adaptation for text classification via distance net-bandits. In AAAI, pages 7830-7838.
S. Helgason. 2011. The radon transform on m. In Integral Geometry and Radon Transforms, pages 1-62. Springer.
Mohammad Rostami. 2021. Lifelong domain adaptation via consolidated internal distribution. Advances in neural in- formation processing systems, 34: 34:11172 11183,
Linqing Huang. Wangbo Zhao, Yong Liu, Duo Yang, Alan Wee-Chung Liew. and Yang You. 2023. An evidential multi-target domain adaptation method based on weighted fusion for cross-domain pattern classification, IEEE Trans- actions on Neural Networks and Learning Systems.
Mohammad Rostami. 2022. Increasing model generalizability for unsupervised visual domain adaptation. In Conference on Lifelong Learning Agents, pages 281-293. PMLR.
Dayuan Jian and Mohammad Rostami. 2023. Unsupervised domain adaptation for training event-based networks using contrastive leaming and uncorrelated conditioning. arXiv preprint X.00v 12424.
Mohammad Rostami and Aram Galstyan. 2023a. Cognitively inspired learning of incremental drifting concepts. In IJ- CAI

---

Mohammad Rostami and Aram Galstyan. 2023b. Overcom- ing concept shift domain-awe aware settings through consol- idated internal distributions. In Proceedings the AAAI conference on artificial intelligence, volume 37, pages 9623-9631.
Yftah Ziser Reichart. 2018. Pivot based language modeling improved neural domain adaptation. Pro- ceedings of NAACL: Human Language Techmologies. Vol- ume 1. pages 1241-1251.
Yftah Ziser and Roi Reichart. 2019. Task refinement learn- ing for improved accuracy and stability of unsupervised domain adaptation. Proceedings of ACT., pages 5895- 5906.
Mohammad Rostami, David Huber, and Tsai-Ching Lu. 2018. A crowdsourcing triage algorithm for geopolitical event forecasting. In Proceedings of the 12th ACM Conference on Recommender Systems, pages 377-381. ACM.
Mohammad Rostami, Soheil Kolouri, Eric Caton, and Kyung- Kim. 2019. Sar image classification using shot cross-domain transfer learning. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 0-0.
Sam Roweis. 1998. Em algorithms for pea and spea. In Proceedings of NeurIPS, pages 626-632.
Sebastian Ruder and Barbara Plank. 2018. Strong baselines for neural remi-supervised learning under domain shift. In Proceedings of ACL). pages 1054.
Minho Ryu, Geonscok Lee, and Kichun Lee. 2022. Knowl- edge distillation bert unsupervised domain adaptation. Knowledge and Information Systems, 64(11):31133312
K. Saito, Y. Ushiku, and Harada. 2018. Asymmetric tri- training for unsupervised domain adaptation. JCMI..
Shai Shalev Shwartz and Shai Ben-David David. 2014. Understand- ing machine learning: From theory to algorithms. Cam- bridge university press.
Jian Shen, Yanru Qu, Weinan Zhang, and Yong Yu. 2018. Wasserstein distance guided representation learning for domain adaptation. AAAI.
Serban Stan and Mohammad Rostami. 2022. Secure domain adaptation with multiple sources. Transactions on Machine Learning Research.
Baochen Sun and Kate Saenko. 2016. Deep coral: Correla- tion aligument for deep domain adaptation. In Luropean conference computer vision, pages 443-450. Springer.
Tatiana Tommasi and Barbara Caputo. 2013. Frustratingly easy nhnn domain adaptation. Proceedings of CVPR, pages 897-904.
Mei Wang and Weihong Deng. 2018. Deep visual domain adaptation: A survey. Neurvcomputing, 312:135-153.
Longhui Wei, Shiliang Zhang, Wen Gao, and Qi Tian. 2018. Person transfer gan to bridge domain gap for person re- identification. In Proceedings of the IEEE Conference on Computer Vision and Pattenn Recognition, pages 79-88.
Dongho Fuzhen Zhuang, Ganbin Zhou, Xisohu Cheng, Lin, and Qing He. 2020, Domain adaptation with category attention network deep sentiment analysis. Proceedings of The Web Conference 2020, pages 3133- 3139.
Werner Zellinger, Thomas Grubinger, Edwin Lughofer, Thomas Natschläger, and Susanne Saminger-Platz. 2017. Central moment discrepancy (emd) for nomain-invariant representation learning. arXiv preprint 1rXiv:1002.088

---

A Sliced Wasserstein distance
Y on the unit sphere as follows:
We relied on minimizing the Sliced Wasserstein (SWD) distance for domain alignment. SWD is de- fined based on the Wasserstein distance (WD) and is mean to come up with a more computationally efficient distribution metric. The WD between two distributions ps and py, is defined as:
SW(ps,Ph
fSr W(Rps(;7), Rpt(s)))dd
(11)
The main advantage of using SWD is that, comput- ing SWD does not require solving a numerically expensive optimization.
/ XxX ((2,))d)(z,
In our practical setting, only samples from the distributions are available and we don't t have the distributional form. Another advantage of SWD is that its empirical version can be computed based the mne-iimessional empirical WD. One- dimensional empirical WD be approximated as the lp dy-distance between the sorted samples. We can compute merely the integrand function in (11) for a known ? and then the integral in (11) via Monte Carlo style numerical integration. To this end, we draw random projection subspace from a uniform distribution that is defined over the unit sphere and then compute ID WD along this sample. We then approximate the integral in (11) by computing the arithmetic average over suitably large enough number of drawn samples. More specifically, the SWD between f-dimensional samples { {Ø(x;)) E R ~ h= and (ø(x?) E R ~ pr) in our setting can be approximated as the following sum:
We(ps. = inf infser(PsTTT)
(8) where PP) is the set of all joint distributions PS,T with marginal single variable distributions ps and PT, and X x Y IR is the cost function, e.g., f2-norm Euclidean distance.
We observe that computing WD involves solv- ing a complicated optimization problem in the gen- eral case. However, when the two distributions are 1 dimenssionnll WD has closed-form solution:
Wolps. S.' (T) (9)
where Ps and Py are the cumulative distributions of the ID distributions ps and pT. This closed- form solution motivates the definition of SWD in order extend applicability of (9) for higher di- mensional distributions.
SWD is defined based on the idea of slice sam- pling (Neal, 2003). The idea is to project two dimensional distributions into their marginal one- dimensional distributions along subspace, i.c., slicing the high-dimensional distributions, and then compute the distance between the two distribution by integrating all the WD between the result- ing ID marginal probability distributions over all possible ID subspaces using the closed form so- lution of WD. This can be a good replacement for the WD as any probability distribution can be expressed by the set of --dimensional marginal projection distributions (Helgason, 2011) 1). More specifically, a one-dimensional slice of the distribu- tion for the distribution ps is defined:
SW (ps:
1
M Σ
(12)
Kn, ((x (m1, olx (())11)))
L.
where n E g is uniformly drawn random sam- ple from the unit f dimmensioonal ball $ and s and [ are the sorted indices of (n ((x;) )=1 for source and target domains, respectively.
We utilize this empirical version of SWD in (12) to align the distributions in the embedding space. Note that the function in (12) is differentiable with respect the encoder parameters and hence we can use gradient- based optimization techniques minimize it with respect to the model parameters.
Rps(t:>) = S (s()))(t (Y,, (r,x)))dz (10)
B Proof of Theorem 1
We use the following theorem by Redko et al. (Redko and Sebban, 2017) and result by Bol- ley (Bolley et al., 2007) on convergence of the em- pirical distribution to the true distribution in terms of the WD distance in our proof.
where 5(-) denotes the Kronecker delta function, ( denotes the vector inner dot product, Sd-1 is the d-dimensional unit sphere, and 7 is the projec- tion direction.
The SWD is defined as the integral of all WD be- tween the sliced distributions over all ID subspaces
Theorem (Redko et al. (Redko and Sebban, 2017)): Under the assumptions described in our

---

framework, assume that a model is trained on the source domain, then for any d' > and ( < V2. there exists a constant number No depending on
both sides of (15) to deduce:
lep - lep-erl =
(h (2,), L(nwo (2/)) ((),0.1)) (16)
d' such that for any 2; > 0 and min(N, M) 2 max(xi
(ri (d' 2).1) with probability at least Bi.
$ (1 -)
the following holds:
Applying (16) in the below, deduce:
eT Scs + W ((i,: iAs) + CS.T
eg + CT = es + eT + ep cp S
(13)
es + ep + le - epl S es + ep + (1 (1-T).
+ +v (2log( 1/9) N + M Bi
(17)
Taking infimum on both sides of (17), we deduce:
Theorem 2 provides an upperbound for the per- formance of source-trained model in the target domain Redko et al. (Redko and Sebban, 2017) prove Theorem 2 for binary classification setting. We also provide our proof in this case but it can be extended.
CS. S Cs, + (1-T).
(18)
Now by considering Theorem 2 for the two do- mains S and 'T and then using (18) in (13), we can conclude:
The second term in Eq. (13) demonstrates the effect of domain shift on the performance of a source-trained model in target domain. When the distance between the two distributions is sig- nificant, this term will be large and hence the up- perbound in Eq. (13) will be loose which means potential performance degradation, Our algorithm mitigates domain gap because this term is mini- mized by minimization of the second and the third terms in Theorem 1.
eT Ses + D(AT, (S) + cs, + +11 -7)
+ / (2log( (2)
1
(19)
N + M
Now using the triangular inequality on the metrics we can deduce:
D(AT, S D(pT,, + D(ps,
S D(pT, + D(ps, (o)) + 2D uP).
(20)
Theorem Consider that we use the procedure described in Algorithm for cross-domain senti- ment analysis, then the following inequality holds for the target expected error:
Now we replace the term D(pp, with its empirical counterpart using Theorem 1.1 in the work by (Bolley et al., 2007).
Theorem 3 (Theorem 1.1 by Bollcy et al. (Bol- ley et al., 2007)): consider that p(.) E P(2) and for some 0. Let f(x) ~ Σ, 8(x;) denote the empirical distribu- tion that is built from the samples { that are drawn i.i.d from æ, ~ p(x). Then for any d' > d and 2; V2, there exists No such that for any 0 and N 2 No max(1, (d" (2),,, we have:
et Sest + D(As. (r) + D(pT, + (1 (1-T)
Sz exp (a:| dpp(x))))))))))))))))))))))))d
+ es + y (2 1/()
+
N (14)
where æ; is constant which depends C(-) and (www)) denotes the expected risk of the optimally joint trained model when used on both the source domain and the pseudo-datased
2 Ne). (21)
((W(p, > e) exp(
2
Proof: Due to the construction of the pseudo- dataset, the probability that the predicted labels for the pseudo-data points to be false is equal to Let:
where W denotes the WD distance. This relation measures the distance between the empirical dis- tribution and the true distribution, expressed in the WD distance.
Applying (20) and (21) on (19) concludes
[C(woo (z(), - C(hwo (z.),
Theorem 2 as stated:
{ 0,
if yf -
(15)
{1. 11 otherwise.
et Ses + D(As. + D(PT.Ap) + (1 - r) + es.p
/ (2 1 1/) N + + 2 No
21
We use Jensen's inequality and take expectation on
(22)

---

Task
B-D
B-=E
B-K
D-+B
D->E
D-*
80/20
82.8 + 0.3
83.2 + 0.5 85.5 + 0.3 78.7 + 0.2 83.3 + 0.2 86.8 + 0.2
90/10
82.9 + 0.5 83.4 + 0.3 85.8 + 0.2 78.5 + 0.4 83.3 + 0.4 86.8 + 0.3
Task
E->B
E-
K
K >B
K-
K E
80/20
78.7 + 0.2 78.5 + 0.5 88.6 + 0.1 76.3 + 0.6 77.9 + 0.4 86.6 + 0.1
90/ 10
78.7 + 0.2 78.0 + 0.4 88.0 + 0.2 76.5 + 0.5 77.3 + 0.3 86.7 + 0.2
Table 3: Effect of label-imbalance on performance.
C
Complexity analysis for GMM estimation
Estimating a GMM distribution usually is compu- tationally expensive tasks. The major reason is that normally the data points are unlabeled. This would necessitate relying on iterative algorithms such ex- pectation maximization (EM) algorithm (Moon, 1996). Preforming iterative E and M steps until convergence leads to high computational complex- ity (Roweis, 1998). However, estimating the mul- timodal distribution with a GMM distribution is much simpler in our learning setting. Existence of labels helps to decouple the Gaussian com- ponents and compute the parameters using MAP estimate for each of the mode parameters in one step as follows:
(a) Source
(b) GMM Dist. Samples
(c) Target: Pre-Adapt.
(d) Target: Post-Adapt.
Figure UMAP visualization for the task D K task in the imbalanced regime of 90% /10% (a) the source domain testing split, (b) the GMM distribution samples, (c) the target domain testing split pre-, and (d) post- domain adaptation. (Best viewed in color).
|
ā N
i = Σ
|S (x:),
Σ
S (Au(at) - is)) (ov(x)) (.))
D Effect of Data Imbalance on Performance
(w) (es))
(23)
Given the above and considering that the source domain data is balanced, complexity of comput- ing ay is O(N) (just checking whether data points af belong to S,). Complexity of computing μ is F/KF/k), where F is the dimension of the embedding space. Complexity of computing the co-variance matrices Σ, is O(F )2). Since, we have & components, the total complexity of com- puting GMM is ((FN))) If O(F) z (k)), which seems to be reasonable practical assumption, then the total complexity of computing GMM would be O(N2). Given the large number of learnable pa- rameters in most deep neural networks which are more than N for most cases, this complexity is fully dominated by complexity of a single step of backpropagation. Hence, this computing the GMM parameters does not increase the computational complexity for.
In practical settings of domain adaptation, the label distribution for the target domain training dataset cannot be enforced to be balanced due to the ab- sence of labels. To study the effect of label im-
balance the performance of the proposed al- gorithm, we synthetically design imbalanced tar- get domain datasets using the Amazon Reviews dataset. We designed two experiments, where the target domain datasets have the 90%/10% and 80%/20% ratios of imbalance between the positive and negative classes, respectively. We have pro- vided domain adaptation results using these two imbalanced scenarios in Table 3 using if-idf fea- tures with d 5000. Comparing Table 3 with Table we observe that performance of our al- gorithm has slightly degraded to relatively simi- lar values for both scenarios. This degradation is expected because the majority class swamps the

---

minority class when generating batches for opti- mization. Note, however, our algorithm has been robust to a large extent with respect to label imbal- ance which demonstrates its practical suitability, when balanced datasets cannot be guaranteed.
For an intuitive sanity check, we have presented the UMAP visualization for the testing split of the task D-K for the scenario 90% / 10% in Figure 4. Observations in Figure 4 match what reported in Table 3, confirming that our algorithm can increase interclass margins when the target domain dataset is imbalanced but dominance of the majority class has led to less separation between the classes for the tasks built using the Amazon reviews dataset.


Reviews:
[{"review_id": "6BnYiCswul", "reviewer": "Reviewer_iCKc", "summary": "", "questions": "(1) There are no references after 2021, raising concerns about the correctness of some claims in this paper. For example, the discrepancy between different domains stems from different usage of terms and phrases (L33), does this domain gap still a major issue in neural models, given NN models can learn context instead of focusing too much on terms and phrases.\nAnother example is BERT features has not been used extensively on domain adaptation tasks (L467), is this claims still true in 2023, as there are no recent references since 2021.\n(2) The figures in the paper are too small, they should be clear by using more space. The paper can make more space from Section 6.1 with a more concise presentation.\n(3) The introduction and related work are mixed together and difficult to understand the contribution of this paper.\n(4) Some citations such as L98-102 are difficult to understand. Using WD for domain alignment boosts performance in visual domains is introduced by Damodaran et al. 2018 or other citations in the same sentence?\n(5) There are some grammar errors such as \"a large enough labeled data points\" in L163.\n(6) This paper reimplements vanilla BERT and reports a different result with the same model in (Du et al., 2020) with a large margin, which raises concerns about the reimplementations.", "limitations": "", "rating": 2, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and effort. We are glad that you have recognized the novelty of our work. We are also grateful that you honestly declared your confidence level as in our experience, many reviewers tend to overestimate their confidence. Given your confidence level and also the opinion of the rest of the reviewers, we would like to respectfully ask you to give us a second chance by elaborating on our response and engaging in post-rebuttal discussions. We also think that if you check the references on UDA for sentiment analysis and compare our work side-by-side, you can evaluate our work more meaningfully. We hope you find our responses below convincing:\n\nThank you for bringing to our attention that we have used works published prior to 2021. This has not been intentional. Upon your point, we did a more expansive search and found the following papers for comparison::\\\n\nLong, Q., Luo, T., Wang, W. and Pan, S., 2022, July. Domain Confused Contrastive Learning for Unsupervised Domain Adaptation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (pp. 2982-2995). (Table 1 and Table 2)\n\nBadr, H., Wanas, N. and Fayek, M., 2022. Unsupervised domain adaptation with post-adaptation labeled domain performance preservation. Machine Learning with Applications, 10, p.100439. (Table 3)\n\nRyu, M., Lee, G. and Lee, K., 2022. Knowledge distillation for bert unsupervised domain adaptation. Knowledge and Information Systems, 64(11), pp.3113-3128. (Table 1)\n\nHuang, L., Zhao, W., Liu, Y., Yang, D., Liew, A.W.C. and You, Y., 2023. An Evidential Multi-Target Domain Adaptation Method Based on Weighted Fusion for Cross-Domain Pattern Classification. IEEE Transactions on Neural Networks and Learning Systems. (Table VI)\n\n\nThere are also several post-2021 published works on “source-free UDA” and “multi-source UDA” for sentiment analysis.  There are also several  2023 unpublished manuscripts on Arxiv. You can easily find these works on Google Scholar. We think that from these works, we can conclude:\n\nUDA has not been resolved in 2023 and still ongoing research continues to address the challenge of the “domain gap” for sentiment analysis. LLMs have made progress to mitigate this challenge but it is not the case that they have solved the challenge completely.\n\nIf you compare our results with the tables, referenced in each work in the above in parentheses, our work remains compelling which demonstrates that our results are still competitive.\n\nWe hope you agree that the ongoing research after 2021 demonstrates that the challenge of UDA for sentiment analysis is still an active research area, despite advances in NN. We will also include the above works in our discussion. If you think there are other missing references, we are more than glad to include them if you give us pointers.\n\n2.  This concern can be easily addressed in the camera version because there will be a 9-page limit and we can easily make the figures larger and nicer. So, we can do this task even without making section 6.1 more concise.\n\n3. If you provide more guidance, we are more than happy to improve the Introduction and the Related Work section given the extra space that we will have. We kindly ask you to provide more details so we know how we can make the paper more clear in these sections and reduce unclarities.\n\n4. We agree we could be more clear and currently that sentence is ambiguous. All those references have explored using WD for domain adaptation in visual domains. We will make that sentence more clear\n\n5. Thank you for pointing out. We will correct it and do a thorough pass to correct grammatical errors and typos. If you find other errors and typos, please let us know.\n\n6. You are correct but we would like to say please first note that our results are close. It is not the case that we are getting totally irrelevant results. Second, the results can depend on hyperparameters and we did our best to find the best hyperparameters to report the best results but our GPU resources are limited and we could not perform an extensive search. We also contacted the authors but unfortunately, we were not given the hyperparameter values used by the authors. In conclusion, we think the disparity is within an acceptable statistical margin. More importantly, please note that as opposed to Du et al, our code is available and you can easily check our implementation. We think that if there are concerns about disparity, our results are much easier for verification because our code is available. In contrast, when the code is not released and the hyperparameters are not provided in a paper, it is not easy to regenerate the results.\n\nWe respectfully ask the reviewer again, to read our responses, check the literature, read the rest of the reviews, engage in post-rebuttal discussion, and if convinced reconsider the final rating. We hope through continued discussions, we can address all the raised concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper presents a novel domain adaptation method and uses experiments to assess its effectiveness. It seems to be effective in tfidf setting, and self-implemented BERT.", "reasons_to_reject": "The writing of this paper makes it difficult to understand.", "questions_to_authors": "(1) There are no references after 2021, raising concerns about the correctness of some claims in this paper. For example, the discrepancy between different domains stems from different usage of terms and phrases (L33), does this domain gap still a major issue in neural models, given NN models can learn context instead of focusing too much on terms and phrases.\nAnother example is BERT features has not been used extensively on domain adaptation tasks (L467), is this claims still true in 2023, as there are no recent references since 2021.\n(2) The figures in the paper are too small, they should be clear by using more space. The paper can make more space from Section 6.1 with a more concise presentation.\n(3) The introduction and related work are mixed together and difficult to understand the contribution of this paper.\n(4) Some citations such as L98-102 are difficult to understand. Using WD for domain alignment boosts performance in visual domains is introduced by Damodaran et al. 2018 or other citations in the same sentence?\n(5) There are some grammar errors such as \"a large enough labeled data points\" in L163.\n(6) This paper reimplements vanilla BERT and reports a different result with the same model in (Du et al., 2020) with a large margin, which raises concerns about the reimplementations.", "paper_topic_and_main_contributions": "This paper presents a domain adaptation (DA) method, which is applied to cross-domain sentiment analysis. The proposed DA method makes contributions in two aspects, the first is aligning distributions of the source domain and target domain in an embedding space, and the second one is inducing larger margins between classes in the embedding space.  The first part is achieved by training an encoder to minimize domain discrepancy with the SWD (Lee et al., 2019) method. The second one is based on learning an intermediate parametric distribution by estimating it as a Gaussian mixture model (GMM).", "excitement": 2, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Enrich references with those published after 2020.", "Typos_Grammar_Style_and_Presentation_Improvements": "See questions."}, "extra_text": {}}, {"review_id": "MlLJBDMNK4", "reviewer": "Reviewer_Sndy", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and effort. We are grateful that recognize the importance of the UDA problem in 2023 and also verify the theoretical soundness of our work. We hope that through continued discussion during the rebuttal period, we can address your concerns and convince you to reconsider your rating.\n\nIt appears to us that by “not convincing based on the experimental performance”, you mean that we are not outperforming all baselines. We would like to bring your attention to a few points:\n\nPlease note that in 5 tasks, we outperformed the baselines using BERT features and could get the best results.\n\nWe know that a major reason for our lower performance in the remaining 7 tasks is that we could not achieve source-only BERT performances similar to Du et al. We are quite certain that if we could achieve similar source-free performance values, our UDA performance could be improved. This outcome is likely because of hyperparameter tuning but unfortunately, the code for Du et al is not public and the authors have not released their code. They also did not provide the hyperparameter values upon our contact. We tried to tune the hyperparameter but our resources are limited. Our code is available and we think regarding this disparity, our implementation is accessible and our reported numbers are much easier to confirm. \n\nIn order to make a judgment about the competence of our method, comparisons should be based on all tasks and experiments. By considering experiments using TF-IDF features and 5 tasks with BERT features, I think we can conclude that our method leads to a convincing performance, and on average, our strengths outweigh the cases for which we are not getting SOTA performance. \n\nThis concern can be easily addressed for the camera-ready version. We are more than happy to reconsider our title based on your feedback. Please recommend the title that might be a better choice and we are more than happy to incorporate your suggestions.\n\nWe agree Fig 3 is very small which is due to the space limit. For review purposes, we made sure that it can be enlarged with good quality so it can be used by the reviewers to make a judgment. Fortunately, there will be a 9th page for the accepted papers and we can easily enlarge that figure to address your concern.\n\nIn summary, we think that concerns 2 and 3 can be addressed straightforwardly. We hope you continue to engage in discussions about concern 1 and we hope upon addressing your concern you reconsider raising your rating because, with the current scores, it looks like your opinion can be very important for the final decision because you have the balancing scoring.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tThe paper is addressing an important problem about domain adaptation in sentiment analysis. The general audience of the conference should be interested in this topic.\n2.\tThe proposed method about maximizing the class distribution gap is theoretically sound.\n", "reasons_to_reject": "1.\tThe proposed method aims at aligning parametric distributions in the embedding space. Authors demonstrate the effectiveness of the proposed method when using tf-idf and pre-trained model BERT. Empirical results show that BERT-based methods outperform tf-idf based approaches. However, when using BERT features, the proposed method can only achieve comparable results compared with BERT-DA. As a result, the proposed method is not convincing based on the experimental performance.\n2.\tThe title of the paper is fairly mis-leading. The idea of the paper is to make use of the re-learning the GMM by sampling high confidence samples, so as to increase the gap between different class distributions. The proposed method does not deal with the representation of the data or feature embedding. The use of \"internal representation\" is confusing.\n3.\tFigure 3 is almost unreadable.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes a Gaussian mixture model (GMM) based method for domain adaptation in sentiment analysis. The idea is to maximize the distribution gap between classes in the GMM. This is achieved by sampling high confidence samples from the GMM to align the source and target domain distribution. Authors conducted experiments on Amazon product review dataset to evaluate the proposed method. Comparison with existing methods shows that the proposed method outperforms baselines when using tf-idf representation, and achieves comparable results when using pre-trained model embedding. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1.\tPage 2, Line 132: It should be X_T instead of X_S. \n2.\tEquation 6 is not balance in braces.\n"}, "extra_text": {}}, {"review_id": "W3BYZwkTTP", "reviewer": "Reviewer_Nxo2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and effort. We are grateful that you have identified that our work is interesting and the theoretical and experimental results are effective and promising. Given your strong rating and high confidence level, we respectfully ask you to engage in post-rebuttal discussions and help the rest of the reviewers converge to your opinion. We respond to your comments below.\n\n\n1. We agree with your comments about LLMs. But as you mentioned the power of LLMs stems from pretraining on a large corpus. In other words, domain shift is less of a challenge when LLMs are used because it is more difficult to find a domain for which LLMs have not been exposed during pre-training. For this reason, this weakness is not specific to our method. Additionally, if we use LLMs in domains where that domain gap exists, we likely will observe performance degradation and UDA still will be relevant.\n\n2. Thank you for pointing out the typos and errors. We will correct them and do a thorough pass to correct other errors and typos.\n\nWe are thrilled that the reviewer has found our work to be strong. We respectfully ask the reviewer to engage in post-rebuttal discussions to help the reviewers with a less confident rating to reach an opinion with higher confidence.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The idea is interesting, in which it uses Gaussian mixture model (GMM) to align different domain distributions and generates pseudo-dataset for a larger margin between the clusters in the source domain.\n2. This paper provides theoretical and empirical analysis for the effectiveness of the proposal.\n3. The proposed method shows promising improvement over the chosen baselines, provided that the results are reproducible when the code is released.", "reasons_to_reject": "The proposed method is mainly effective when using the traditional text features like tf-idf, by mitigating the domain gap. But when employing the pretrained language model like BERT and large language model (LLM), which are pretrained in large scale or extremely large corpus and performs strong domain transfer capability, this proposed method may be less competitive compared to them. The experiments in the paper prove the limitation.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a method for cross-domain sentiment analysis based on aligning two domain-specific distributions in a shared embedding space and inducing larger margins between the classes using an intermediate multi-modal GMM distribution to mitigate the \"domain gap\" between different applications. \nThe contribution is reflected in that learn a parametric distribution for the source domain in a cross-domain embedding space by estimating  the distribution as a Gaussian mixture model (GMM) and then use the GMM distribution to align the source and the target distributions using confident samples, drawn for them GMM to increase the interclass margins to reduce the domain gap.\nThis paper provides theoretical and empirical analysis for the effectiveness of the proposal.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Some captions of certain tables like Table 2 are too simple, which makes it difficult to read.\n2. #line 938, there is a typos error as you write \"Consider that we the procedure\"."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a great theoretical solution and theories and proofs are well explained, however, in practice the results are comparable or superior only with traditional encoding methods like tf-idf, other results where more advanced encoding methods like BERT are used other approaches mostly outperform the proposed methods. 

Although the proposed method is used in 4 different cross-domains it is still done on a small set of benchmarked datasets, hence, it makes the effectiveness of the method questionable. Sentiment is very subjective and such a theoretical method should be tested on a big dataset instead.
Decision: Accept-Findings