Learning a Max-Margin Classifier for Cross-Domain Sentiment Analysis
Authors: Mohammad Rostami
Aram Galstyan
Venue:  (2021)
Keywords: natural language processing
sentiment analysis
cross-domain data representation
distribution alignment

Abstract:  Sentiment analysis is a costly yet necessary task for enterprises to study the opinions of their costumers to improve their products and services and to determine optimal   marketing strategies. Due to existence of a wide range of domains across different products and services, cross-domain sentiment analysis methods have received significant attention in recent years. These methods mitigate the domain gap between different applications by training cross-domain generalizable classifiers which help to relax the need for individual data annotation per each domain. Most existing methods focus on learning domain-agnostic representations that are invariant with respect to both the source and the target domains. As a result, a classifier that is trained using  annotated data in a source domain, would generalize well in a related target domain. In this work, we introduce a new domain adaptation method which induces large margins between different classes in an embedding space based on the notion of prototypical distribution. This embedding space is trained to be domain-agnostic by matching the data distributions across the domains. Large margins in the source domain help to reduce the effect of ``domain shift'' on the performance of a trained classifier in the target domain. Theoreticaland empirical analysis are provided to demonstrate that the method is effective. 

Under review as a conference paper at ICLR 2021
LEARNING A MAX- CLASSIFIER FOR CROSS- DOMAIN SENTIMENT ANALYSIS
Anonymous authors Paper under double-blind review
ABSTRACT
Sentiment analysis is a costly yet necessary task for enterprises to study the opin- ions of their customers to improve their products and services and to determine optimal marketing strategies. Due to the existence of wide range of domains across different products and services, cross-domain sentiment analysis methods have received significant attention recent years. These methods mitigate the domain gap between different applications by training cross-domain generalizable classifiers which help to relax the need for individual data annotation for each domain. Most existing methods focus on learning domain-agnoste representations that are invariant with respect to both the source and the target domains. As a result, classifier that is trained using annotated data in source domain, would generalize well in a related target domain. In this work, we introduce a new domain adaptation method which induces large margins between different classes in an embedding space based on the notion of prototypical distribution. This embedding space is trained be comain-agnosste by matching the data distributions across the domains. Large margins in the source domain help to reduce the effect of "domain shift" on the performance of trained classifier the target domain. Theoretical and empirical analysis are provided to demonstrate that the method is effective.
INTRODUCTION
The main goal in sentiment classification is to predict the polarity of users automatically after collecting their feedback, c-g., Amazon customer reviews. Popularity of online shopping and reviews, fueled further by the recent pandemic, provides valuable resource for businesses to study the behavior and preferences of consumers and to align their products and services with the market demand. A major challenge for automatic sentiment analysis is that polarity is expressed using completely dissimilar terms and phrases in different domains. For example, while terms such as "fascinating" and "boring" are used to describe books, terms such "tasty" and "stale" are used to describe food products. As a result of this discrepancy, a model that is trained for a particular domain may not generalize well in other different domains, referred as the problem of "domain gap" (Wei et al. (0188). Since generating annotated training data for all domains is expensive and time-consuming, cross-domain sentiment analysis has gained significant attention recently (Saito et al.
2018 Li et al. 2017; Peng et al.) 2018 He et al. 2018 Li et al.) 2018 Barnes et al. 2018) Sarma et al. 2019 Li et al. 2019 Guo al. 2020 Xi et al. 2020 Dai et al.) [0200] Lin al.
2020). The goal in cross-domain sentiment classification is to relax the need for data annotation via transferring knowledge from another domain with annotated data to domains with unannotated data.
The above problem has been studied more broadly in the "domain adaptation" literature. A common approach for domain adaptation is map data points from two domains into a shared embedding space to align the data distributions (Redko & Sebban 2017). Since the embedding space would become domain-agnoste generalize in the target domain. In the sentiment analysis problem, this means that polarity of natural language can be expressed independent of the domain in the embedding space. We can model this embedding space the output of shared deep encoder which is trained to align the distributions of both domains at its output. This training procedure have been implemented using both adversarial learning (Pei et al. 2018 Long et al. 2018 Li et al.) 2019 Dai et al. 2020), which aligns distributions indirectly, or by loss functions that are designed to directly align the two distributions (Peng et al. 2018 [Barnes al. 2018 Kang al. 2019 Guo et al. 2020 Xi et al. 2020 Lin et al. 2020).
i.e., a classifier that is trained using the source domain annotated data, will

---

Under review as a conference paper at ICLR 2021
Contributions: our main contribution is to develop new cross dross-domain sentiment analysis algorithm for model adaptation by introducing large margins between classes in the source domain. Our idea is based learning prototypical distribution for the source domain in cross- embedding space which is trained to be domain-agnostic. We model this distribution as a Gaussian mixture modal (GMM). We estimate the parameters of the prototypical distribution using a subset of source samples for which the classifier is confident about its predictions. As a result, larger margins between classes are introduced in the prototypical distribution which help reducing domain gap. We then use this prototypical distribution to align the source and the target distributions via minimizing the Sliced Wasserstein Distance (SWD) (Lee et al., 2019). We draw confident random samples from this distribution and enforce the distribution of the target in the embedding matches this prototypical distribution in addition to the source distribution. We provide theoretical proof to demonstrate that our method minimizes an upperbound for the target domain expected error. Experimental results demonstrate that our algorithm outperforms state-of-the-ar sentiment analysis algorithms.
2 RELATED WORK
While domain adaptation methods for visual domains usually use generative adversarial networks (GANs) (Goodfellow et al., 2014) and align distributions indirectly, the dominant approach for cross-domain sentiment analysis is to design appropriate loss functions that directly impose domain alignment. The main reason is that natural language is expressed in terms of discrete values such as words, phrases, and sentences. Since this domain is not continuous, even if we convert natural language into realevalued vectors, cannot be easily implemented for pure natural language processing (NLP) applications. Several alignment loss functions have been designed for cross-domain sentiment analysis.
not differentiable. Hence, adversarial learning procedure
group of methods are based on aligning the lower-order distributional moments, c.g., means and covariances, across the two domains, in an embedding space (Wu & Huang 2016: Peng et al. 20188] Sarma et al.) probability distribution metries to consider the encoded information in higher order statistics (Shen et al.] 2018). Damodaran et al. (Bhushan Damodaran et al.) 2018) demonstrated that using Wasserstein distance (WD) for domain alignment boosts the performance significantly in visual domain applications (Long et al. 2015 Sun & Saenko 2016). In the current work, rely on the sliced Wasserstein distance (S (SWD) for aligning distribution. SWD has been used for domain adaptation in visual domains (Lee et al. 2019).
2019. Guo et al. 2020p. An improvement over these methods
The major reason for performance degradation of source-trained model in target domain stems from "domain shift", i.e., the boundaries between the classes change in the embedding space even for related domains which in turn increases possibility of misclassification that if a max mar-in classifier is trained in the source domain, it can generalize better than many methods that try to align distributions without further model adaptation (Tommasi & Caputo 2013). Inspired by the notion of "class prototypes", our method is based on both aligning distributions in the embedding space and also inducing larger margins between classes using the notion of "prototypical distributions". Recently, cross-domain alignment of the class prototypes has been used for domain adaptation (Pan et al. 2019: Chen et al. 2019). The idea is that when a deep network classifier is trained in domain with annotated data, data points of classes form separable clusters in an embedding space, modeled via network responses in hidden layers. A class prototype is defined as the mean of each class-specific data cluster in the embedding space. Domain adaptation then can be addressed by aligning the prototypes across the two domains as a surrogate for distributional aligmment. Following the above, our work is based on using the prototypical distribution, rather simply the prototypes, to induce maximum margin between the class-specific clusters after an initial training phase in the source domain. Since the prototypical distribution a multimodal distribution, we can estimate it using Gaussian mixture model (GMM). We estimate the GMM using the source sample for which the classifier is confident and use random samples with high-confident labels to induce larger margins between classes, compared to using the original source domain data.
has been argued
3 CROSS-DOMAIN SENTIMENT ANALYSIS
Consider two sentiment analysis problems in source domain S with annotated dataset Ds (Xs, Ys). where Xs = [zi...... xiyl E X C RdxN and Ys = "y..., y% E y C R and
target domain T with unannotated dataset Dr (Xs), where X [mi....,,',, xyy E x C
2

---

Under review as a conference paper at ICLR 2021
Source Domain data
Source Domuin
Data
Embedding Space
Embedding Space
Encoding
Encoding
Laraer
RAkouario, Recefiturh. tedkeeilags
Emccoles
Classtte
Mdmouttie, MapolWuull.. Entweliiggs
Encoder
Margjaa
Clasiii
Ditribution Sucaple
1. The model pre-trained the source domain
GMM is estimated in the embedding space
3. The model adapted the target using both GMM samples and source data
targetdomain domain
Target Dumain data
Embedding Space
Embedding Space
Domain Shift Caused by Domain Gap
Domain Shift effect is mitigated due to Larger Margins
Post-Adaptation
Targel Domain
Poor Generalization Due to Small Margins
Improved Generalization
Figure I: Architecture of the proposed cross-domain sentiment analysis framework. Left: separable clusters formed in the embedding space after initial supervised model training in the source domain and then the prototypical distribution estimated as a GMM. Right: random samples from the GMM with high-confident labels used to generate a pseudo-dataset. The pseudo-dataset induces larger margins between the classes to mitigate the effect domain shift the target domain.
RdxM The real-valued feature vectors Xs and XT are obtained after pre-processing the input text data using common NLP methods, e.g., bag of words or word2vec. We consider that both domains share the same type of sentiments and hence the one-hot labels yº encode i sentiment types, e-g.. negative positive in binary sentiment analysis. Additionally, we assume that the source and the target feature data points are drawn independent and identically distributed from the domain-specific distributions æf ~ ps(a) (z) and at (r()), respectively, where pr(x)) # ps(x).
Given a family of parametric functions fe R2d + V, e.g., deep neural networks with learnable parameters 9, and considering an ideal labeling function f()). e.g., V(x, y) y = f(x). the goal is to search for the optimal predictor model fe- (-) in this family for the target domain. This model should have minimal expected error for sentiment analysis, i.e., 0" arg ming (ca) = arg ming ( (x:) (C(((")) Jo (xx")) where L(-) is proper loss function and B(.) denotes the expectation operator. Since the target domain data unlabeled, the naive approach is to estimate the optimal model using the standard empirical risk minimization (ERM) in the source domain:
i arg min(éo Ys, C)} arg min #
Σ ((fo(=)).)))
Given large enough labeled datasets in the source domain, ERM model generalizes well in the source domain. The source-trained model may also perform much better than chance in the target domain, given cross-domain. However, its performance will degrade the target domain compared to its performance in the source domain due to existing distributional discrepancy between the two domains, since Ps A Pr. Our goal benefit from the encoded information in the unlabeled target domain data and adapt the source-traines classifier fa to generalize in the target domain. We use the common approach of reducing the domain gap by mapping data into a shared embedding space.
We consider that the predictor model fo(-) can be decomposed into a deep encoder (,(+): x 2
IRP and classifier (n 2 y such that fo = hww o &e. where 0 (w, (w) 2 is embedding space which is modeled by the encoder responses at its output. We assume that the classes have become separable for the source domain in this space after an initial training phase (see Figure [ left). If we can adapt the source-trained encoder network such that the two domains share similar distributions Z, i.e., (-) z3 ((pr)()), the embedding space would become domain camain-agnostic. As result, the source-trained classifier network will generalize with similar performance in the target domain. number of prior cross-dee doms-nn sentiment analysis algorithms use this strategy, select a proper probability distribution metric to compute the distance between the distributions ppps (x=))
3

---

Under review as a conference paper at ICLR 2021
and ((pr(x))) and then train the encoder network to align the domains via minimizing this distance:
ü, の arg min 0,30
Σ ((ww(dd(z))) uf) (0)))
where -) denotes a probability metric measure the domain discrepancy and À is trade-off parameter between the source ERM and the domain alignment term. In this work, we use SWD (Lee)
al. 2019) compute -) in equation 2 Using SWD has three advantages. First, SWD can be computed efficiently compared to WD based on closed form solution of WD distance in 2D. Second, SWD can be computed using the empirical samples that are drawn from the two distributions. Finally, SWD possesses non-vanishing gradient even when the support of the two distributions do not overlap (Bonnotte| 2013 Lee et al. 2019). Hence SWD is suitable for deep learning problems which are normally sol ved using first-order gradient-based based optimization techniques, e.g., Adam.
While methods based on variations of equation 2 are effective to reduce the domain gap to some extent, our goal is to improve upon the baseline obtained by equation 2 by introducing loss term that increases the margin between classes in the target domain (check the embedding space in Figure [ right). By doing so, our goal is mitigate the negative effect of domain shift in the target domain.
LEARNING MAX-MARGIN CLASSIFIERS
Our idea for increasing margins between the classes is based on using an intermediate proto- typical distribution in the embedding space. We demonstrate that this distribution can be used to induce larger margins between the classes. To this end, we consider that the classifier subnet- work consists of softmax layer. This means that the classifier should become maximum posteriori (MAP) estimator after training to be able to assign membership probability to a given input feature vector. Under this formu- lation, the model will generalize in the source domain if after supervised training of the model using the source data, the input distribution is transformed into a multi-modal distribution p-(+)= o (ps) with & modes in the embed- ding space (see Figure [ left). Each mode of this distribution corresponds to one type of senti- ments. The mean for each of these modes corre- sponds the notion of "class prototype" in the prior works (Pan et al. 2019 Chen et al. (019)). Following a similar terminology, we refer to this distribution as the prototypical distribution. The geometric distance between the modes of proto- typical distribution corresponds to the margins between classes. we test the source-trained model in the target domain, the boundaries between class modes will change due to the existence of "domain shift", i.e., de (PT) & (ps)))),, Intu- itively, as visualized in Figure[] if we can increase the margins between the class-specific modes in the source domain, domain shift will cause less performance degradation TTommaii & Caputo 2013).
Algorithm SAUM (A,
1: Initial Training:
2: Input: source dataset Ds (Xs, Ys),
3: Training on the Source Domain:
4: O (ıog, (0)
5:
arg ming Σ, C(fo(=)),
6: Prototypical Distribution Estimation:
7: Use equation 4 Had estimate ojj Pú; Y,j
8: Model Adaptation:
9: Input: target dataset DT = XT
10:
Pseudo-Dataset Generation:
11:
Di (Zp, Yp)
12:
([=P, (2(2). S SN
P y 1), where:
13:
14:
V. arg max, (hrnn (=))
15:
max Aigg (z(") >
for itr 1, ..... ITR do
17:
draw data batches from Ds. Dy, and Do
Update the model by solving equation 6
19:
end for
We estimate the prototypical distribution in the embedding space as a parametric GMM as follows:
p)())= = Σ (2]((,), E,),
(3)
where μ4 and Σ, denote the mean and co-variance matrices for each component and a, denotes mix- ture weights for each component. We need to solve for these parameters to estimate the prototypical distribution. Note that unlike usual cases in which expectation maximization algorithm (Bilmes et al.
4

---

Under review as a conference paper at ICLR 2021
1998) is used to estimate GMM parameters, the source data points are labeled. As result, can estimate μ4 and Σ for each component independently using standard MAP estimates. Similarly, the weights aj be computed by MAP estimate. Let Sj denote the support set for class in the training dataset, i.e., Sj { yf) E 'Dsl arg max yf = j}- To cancel out outliers, we include only those source samples in the S3 sets, for which the source-trained model predicts the corresponding labels correctly. The MAP estimate for the mode parameters can be computed as:
S 6 = N (0) (EEE,
B = Σ
Σ, Σ
SS (or((x)) - (2))) (0)(()))) - (.)).
SS
(0) MEEE,
(4)
Computations in Eq. equation 4 can be done efficiently. For complexity analysis, please refer to the Appendix. Our idea is to use this prototypical distributional estimate to induce larger margins in the source domain (see Figure [ right). We update the domain alignment term in equation to induce larger margins. To this end, we update the source domain samples in the domain alignment term by samples of a labeled pseudo-dataset Dp (Zp. Yp) that we generate using the GMM estimate, where Zp = [ 2,,,11 E IRPXNp Yp- y .... Vx E RR This pseudo-dataset is generated using the prototypical distribution. We draw samples from the prototypical distributional estimate z. ~ (z) for this purpose. To induce larger margins between classes, we feed the initial drawn samples into the classifier network and check the confidence level of the classifier about its predictions for these randomly drawn samples. We set a threshold 0 < T < level and select a subset of the drawn samples for which the confidence level of the classifier is more than
(z,' y²) E Dr if z! ~ (z(z) and max()(²?)) > and yf = arg max (h(2?)) (5)
Given the GMM distributional form, selection of samples based on the threshold 7 means that we include GMM samples that are closer to the class prototypes (see Figure IT. In other words, the margin between the clusters in the source domain increase if we use the generated pseudo-dataset for domain alignment. Hence, we update equation 2 and solve the following optimization problem:
1 N Σ
è, Φ arg min w, N
L (hw
N,
(6)
AD Xp)) + (x())
The first and the second terms in equation are ERM terms for the source dataset and the generated pseudo-dataset dataset to guarantee that the classifier continues to generalize well in the source domain after adaptation. The third and the fourth terms are empirical SWD losses (see Appendix for more details) that align the source and the target domain distributions using the pseudo-dataset which as we describe induces larger margins. The hope is that as visualized in Figure [] these terms can reduce the effect of domain shift. Our proposed solution, named Sentiment Aanlysis Using Max-Margin classifiers (SAUM²), is presented and visualized in Algorithm I and Figure respectively.
5 THEORETICAL ANALYSIS
We provide a theoretical justification for our algorithm. Following a standard PAC-Icarning framework. we prove that Algorithm I minimizes an upperbound for the target domain expected error. Consider that the hypothesis class in a PAC-learning setting is the family of classifier sub-networks H (-)m(h))) : 2 + R. E VR,, where V denotes the number of learnable parameters. We represent the expected error for a model (.) E H on the source and the target domains by es(wr) and (w)), respectively. Given the source and the target datasets, we can represent the empirical source and target distributions in the embedding space as AS = N Σn' (()v(=))) and jur = 1. EM 5( (1 Similarly, we can build an empirical distribution for prototypical M7
distribution ΩØ 2, N, "," 8(2g). In our analysis we also use the notion of joint-optimal model h.) in our analysis which is defined as: w" = arg min es.7 = arg min (es + er} for any given domains S and T. When have labeled data in both domains, this is the best model that can be trained using ERM. Existence a good joint-trained model guarantees that the domains are related, e.g., similar sentiment polarities are encoded consistently across the two domains.
5

---

Under review as a conference paper at ICLR 2021
Theorem 1: Consider that we the procedure described in Algorithm I for cross cross-domain sentiment analysis, then the following inequality holds for the target expected error:
et Ses + D(ūs, + D(AT,(AA + (1 (1- r) es. + V (2 (o00(t)/91111 +
1 M
+
2
1 N
where & a constant number which depends on characteristic of the loss function C(-).
Proof: The complete proof is included in the Appendix due to space limit.
Theorem 1 provides explanation to justify Algorithm [ We observe that all the terms in the upperbound of the target expected error in the right-hand side of equation 7 are minimized by SAUM" M algorithm. The source expected error is minimized as the first term in equation 6 The second and the third terms terms are minimized as the third and fourth terms of equation 6 The fourth term 1 T will be small if we set T 1. Note however, when we select 7 too close to 1, the samples will be centered around the prototypes. As a result, we will not match the higher-order distributional moments in the terms D(js,, D(p) and D(ss, (TT) and this can make the upperbound looser. The term ES,P P is minimized through the first and the second term of equation This is highly important as using the pseudo-dataset provides a way to minimize this term. As can be seen in our proof in the Appendix, if we don't 't use the pseudo-daten cannot be minimized directly due to lack of having annotated data in the target domain. The last term in equation is is constant term that as common in PAC-learning can become negligible states that in order train good model if we have access large enough datasets. Hence all the terms in the upperbound are minimized and if this upperbound is tight, then a good model will be trained for the target domain. If the two domain are related, c.g., share the same classes, and also classes become separable in the embedding space, i.e., GMM estimation error is small, then the upperbound is going to be likely tight. However, we highlight that possibility of tight upperbound a condition for our algorithm to work. This is a common limitation for most parametric algorithms.
this terms is replace with es,7 which
6 EXPERIMENTAL VALIDATION
Our implemented code is available at Appendix
6.1 EXPERIMENTAL SETUP
Most existing works report performance cross-domain tasks that are defined using the Amazon Reviews benchmark dataset (Blitzer et al.) 2007). The dataset is built using Amazon product reviews from four product domains: Books (B), DVD (D), Electronics (E), and Kitchen (K) appliances. Each review is considered to have positive (higher than 3 stars) or negative (3 stars or lower) sentiment. Each Review is encoded in a 5000 dimensional or 30000 dimensional If-idf feature vector bag-of- words unigrams and bigrams. We report performance on the 12 definable cross-domain tasks for this dataset. Each task consists of 2000 labeled reviews for the source domain and 2000 unlabeled reviews for the target domain, and 2500-5500 examples for testing. We report the average prediction accuracy and standard deviation (std) over 10 runs on the target domain testing split for our algorithm.
We compare our method against several recently developed algorithms. We compare against DSN (Bousmalis et al. 2016) CMD /Zellinger et al. (2017). ASYM (Saito et al. 22118), PBLM (Ziser & Reichart 20188) MT-Tri (Ruder & Plank] 2018). TRL (Ziser & Reichart 2019). and TAT (Liu et al. 2019). DSN and CMD are similar to SAU M in that both align distributions in an embedding space. DSN learns shared and domain speciifc knowledge for each domain and aligns the shared knowledge using the mean-based maximum mean discrepancy metric. CMD uses the central moment discrepancy metric for domain alignment. ASYM benefits from the idea of labeling of the target samples to updated the base model. MT-Tri is based on YSYM but it also benefits from multi-task learning. TRL and PBLM do not use distribution alignment and are based on the pivot based language model. TAT is a recent work that has used adversarial learning successfully for cross-domain sentiment analysis. We provided results by the authors for the tasks in our table. We report std if std is reported in the original paper. All the methods except TAT that uses 30000 dimensional features use 5000 dimensional features. In our results, methods are comparable they use features with the same dimension. We report performance of the source only (SO) model lowerbound.
6

---

Under review as a conference paper at ICLR 2021
Task
B-+D
B B
B-*K
D-+B
D-E
D-*
SO
81.7 0.2
74.0 + 0.6
76.4 = 1.0
14.5:03.
75.6 0.7
79.5 0.4
DSN
8
0.5
0.6
80.1
81.4 1.1
0.7
CMD
82.6 0.3
0.6
0.3
80.7
8.5
84.8 84.8
ASYM
80.7
79.8
82.5
73.2
77.0
82.5
PBLM
84.2
77.6
82.5
82.5
79.6
83.2
MT-Tri
81.2
78.0
78.8
77.1
81.0
79.5
TRL
82.2
82.7
SAUM
83.2
0.2
83.9 1 0.3
85.9 1 0.3
80.3 1 0.4
84.2 1 0.3
87.3 + 0.2
TAT
84.5
80.1
83.6
81.9
81.9
84.0
2
86.2 + 0.2
85.1 1: 0.2
87.6 1 0.2
80,9 1 0.5
85.2 1 0.2
88.6 + 0.2
Task
E-+B
E-+D
E-K
K-+B
K-D
K-+
SO
72.3 + 1.5
74.2 1 0.6
85.6 + 0.6
73.1 + 0.1
75.2 1 0.7
85.4 1 1.0
DSN
75.1 + 0.4
77.1 1: 0.3
87.2 1 0.7
76.4 1 0.5
78.0 1 1.4
86.7 1 0.7
CMD
74.9 1 0.6
77.4 0.3
86.4
75.8
77.7 0.4
86.7 1
ASYM
73.2
72,9
86.9
72.5
74.9
84.6
PBLM
71.4
75.0
87.8
74.2
79.8
87.1
MT-Tri
73.5
75,4
87.2
73.8
77.8
86.0
TRL
75.8
72.1
SAUM
78.6 + 0.4
79.7 + 0.2
89.2 + 0.2
76.7 0.4
79.1 4: 0.4
87.0 + 0.1
TAT
83.2
77.9
90.0
75.8
77.7
88.2
2
78.8 +:
78.9 0.3
90.1
+
78.1 d: 0.2
78.8 0.4
88.1 0.1
Table 1: Classification accuracy for the cross-domain sentiment analysis tasks for Amazon Reviews dataset. this table, denotes methods that use 30000 dimensional rf-idf feature vectors.
used the benchmark network architecture that is used in the above mentioned works. We used an encoder with one hidden dense layer with 50 nodes with sigmoid activation function. The classifiers consist softmax layer with two output nodes. We implemented our method in Keras, used adam optimizer, and tuned the learning in the source domain. We 7 0.99 and À 10-2, We observed empirically that our algorithm is not sensitive to the value of X.
6.2 RESULTS
Our results are reported in Table I In this table, bold font denotes best performance among the methods that use 5000 dimensional features. We see that M algorithm performs reasonably well and in most cases leads to the best performance. Note that this is not unexpected as none of the methods has the best performance across all tasks. We observe from this table that overall the methods DSN and CMD which are based aligning the source and target distributions- which are more similar to our algorithm- have relatively similar performances. This observation suggests that should not expect considerable performance boost if we simply align the distributions by designing new alignment loss function. This means that outperformance
SAU A compared to these methods likely stems from inducing larger margins. We verify this intuition in our ablative study. We also observe that increasing the dimensional of if-idf features to 30000 leads to performance boost which probably the reason behind good performance TAT compared the rest methods. Hence, we need to use the same dimension for features for fair comparison among the methods.
To provide intuition for the rationale used, we have used UMAP Mcinnes al. (2018) visualization tool to reduce the dimension of the data representations in the 50D embedding space to two for the purpose of 2D visualization. Figure visualizes the testing splits of the source domain before model adaptation, the testing splits of the target domain before and after model adaptation, and finally random samples drawn from the prototypical distribution for the D->K task. Each point represents one data point and each color represents one of the sentiments. Observing Figure 2a and Figure 2b) we conclude that GMM prototypical distribution approximates the source domain distribution reasonably well and at the same time, margin between the classes in the boundary region observable. Figure 2c visualizes the target domain samples prior to model adaptation. As expected, we observe that domain gap has caused less separations between the classes, also evident from SO performance in Table Figure 2d visualizes the target domain samples after adaptation using S AU 2 algorithm. Comparing Figure with Figure 2nd Figure 2a that the classes

---

Under review as a conference paper at ICLR 2021
(a) Source
(b) Proto. Dist. Samples (c) Target:
Pre-Adapt. (d) Target: Post- fost-Adapt.
Figure 2: UMAP visualization for the task D task: (a) the source domain testing split, (b) the prototypical distribution samples, (c) the target domain testing split prior to adaptation, and (d) the target domain testing split after adaptation. (Best viewed in color).
Table 2: Performance AO setting.
83.0
Task
B-D
B-EE
B-*K
Accuracy
81.9 + 0.5
80.9 + 0.8
83.2 - 0.8
Task
D-+B
D->E
D->K
02.0
74.0 + 0.9
80.9 0.7
83.4 = 0.6
82.4
Task
E-+B
E-+D
E-*K
74.1 + 0.6
74.0 + 0.3
87.8 + 0.9
Tau
Task
D-BB
D->E
D->K
Figure 3: Effect of on performance.
74.1 + 0.6
74.0 + 0.3
87.8 = 0.9
have become more separated. Also, careful comparison of Figure 2d and Figure reveals SAU A1 algorithm has led to bias in the target domain to move the data points further from the boundary.
6.3 ABLATION STUDIES
First note that the source only (SO) model result, which is trained using equation [ already serve basic ablative study to verifyy the effect of domain alignment. Improvement over this baseline demonstrates effect of using the information which is encoded in the target unlabeled data.
In Table 2, have provided an additional ablative studies. We have reported result of alignment only (AO) model adaptation based on equation 2 The AO model does not benefit from the margins that SAUM algorithm induces between the classes. Comparing AO results with Table 1 we conclude that the effect of increased margins is important in our performance. Compared to other cross-domain sentiment analysis methods, the performance boost for algorithm stems from inducing large margins. This suggests that researchers may check to investigate secondary techniques for domain adaptation in NLP domains, in addition probability distribution alignment.
Finally, we have studied the effect of the value of the confidence parameter on performance. In Figure 3 we have visualized the performance of our algorithm for the task - D when 7 varied in the interval [0, 0.99]. When that as we increase the value of r, the performance increases as a result of inducing larger margins. For values > 0.8, the performance has less variance which suggests robustness of performance if T 1. These empirical observations about 7 accord with our thcoretical result, stated equation 7
0. the samples are not necessarily confident samples. We observe
CONCLUSIONS
We developed a method for cross-domain sentiment analysis based on aligning two domain specific distributions in a shared embedding space. We demonstrated that one can improve upon this baseline by inducing larger margins between the classes in the source domain using an intermediate multi- modal prototypical distribution. As result, the domain shift problem is mitigated in the target domain. Our experiments demonstrate that our algorithm is effective. A future research direction is to address cross-domain sentiment analysis when different types of sentiment exists across the domains.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Jeremy Barnes, Roman Klinger, and Sabine Schulte im Walde. Projecting embeddings for domain adaption: Joint modeling of sentiment analysis diverse domains. In Proceedings of the 27th International Conference Computational Linguistics, pp. 818-830, 2018.
Bharath Bhushan Damodaran, Benjamin Kellenberger, Rémi Flamary, Devis Tuia, and Nicolas Courty. Deepjdot: Deep joint distribution optimal transport for unsupervised domain adaptation. In Proceedings of the European Conference Computer Vision (ECCV), pp. 447-463, 2018.
Jeff A Bilmes et al. A gentle tutorial of the em algorithm and its application parameter estimation for gaussian mixture and hidden markov models. International Computer Science Institute, 4(510): 126, 1998.
John Blitzer, Mark Dredze, and Fernando Pereira. Biographies, bollywood, boom- boom-boxes and blenders: Domain adaptation for sentiment classification. In Proceedings of the 45th annual meeting of the association of computational linguistics, pp. 440-447, 2007.
François Bolley, Arnaud Guillin, and Cédric Villani. Quantitative concentration inequalities for empirical measures non-compact spaces. Probability Theory and Related Fields, 137(3-4): 541-593, 2007.
N. Bonnotte. Unidimensional and evolution methods for optimal transportation. PhD thesis, Paris 11, 2013.
Konstantinos Bousmalis, George Trigeorgis, Nathan Silberman, Dilip Krishnan, and Dumitru Erhan. Domain separation networks. In Advances in Neural Information Processing Systems, pp. 343-351, 2016.
Chaoqi Chen, Weiping Xie, Wenbing Huang, Yu Rong, Xinghao Ding, Yue Huang, Tingyang Xu, and Junzhou Huang. Progressive feature alignment for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 627-63676666 2019.
Yong Dai, Jian Liu, Xiancong Ren, and Zenglin Xu. Adversarial training based multi-source unsupervised domain adaptation for sentiment analysis. In AAAI, pp. 7618-7625, 2020.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Faries Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems, pp. 2672-2680, 2014.
Han Guo, Ramakanth Pasumuru, and Mohit Bansal. Multi-source domain adaptation for text classifi- cation via distance net-bandits. In AAAI, pp. 7830-7838. 2020.
Ruidan He, Wee Sun Lee, Hwee Tou Ng, and Daniel Dahlmeier. Adaptive semi-supervised learning for cross-domain sentiment classification. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 3467-3476, 2018
S. Helgason. The radon transform on m. In Integral Geometry and Radon Transforms, pp. 1-62. Springer, 2011.
Guoliang Kang, Lu Jiang, Yi Yang, and Alexander G Hauptmann. Contrastive adaptation network for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4893 4902, 2019.
Chen-Yu Yu Lee, Tanmay Batra, Mohammad Haris Baig, and Daniel Ulbricht. Sliced wasserstein discrepancy for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 10285-10295, 2019.
Zheng Li, Yun Zhang, Ying Wei, Yuxiang Wu, and Qiang Yang. End-to-end adversarial memory network for cross-domain sentiment classification. In IJCAI, pp. 2237-2243, 2017.
Zheng Li, Ying Wei, Yu Zhang, and Qiang Yang. Hierarchical attention transfer network for cross- domain sentiment classification. Thirty-Secona AAAI Conference on Artificial Intelligence. 2018.

---

Under review as a conference paper at ICLR 2021
Zheng Li, Xin Li, Ying Wei, Lidong Bing, Yu Zhang, and Qiang Yang. Transferable end-to-end aspect- based sentiment analysis with selective adversarial learning. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNI.P-ICNIIP),, pp. 4582-4592, 2019.
Chuang Lin, Sicheng Zhao, Lei Meng, and Tat-Seng Chua. Multi-source domain adaptation for visual sentiment classification. AAAI, pp. 2661-2666 2020.
Hong Liu, Mingsheng Long, Jianmin Wang, and Michael Jordan. Transferable adversarial training: A general approach to adapting deep classifiers. In International Conference on Machine Learning. pp. 4013-4022, 2019.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In Proceedingy of International Conference on Machine Learning, pp. 97-105, 2015.
Mingsheng Long, Zhangjie Cao, Jianmin Wang. and Michael Jordan. Conditional adversarial domain adaptation. In Advances in Neural Information Processing Systems, pp. 1640-1650, 2018.
Leland McInnes, John Healy, Nathaniel Saul, and Lukas Großberger. UMAP: Uniform manifold approximation and projection. Journal of Open Source Software, 3(29):661, 2018.
Todd K Moon. The expectation-maxxiiiztion algorithm. IEEE Signal processing magazine, 13(6): 47-60, 1996.
R. Neal. Slice sampling. Annals of statistics, pp. 705-741, 2003.
Yingwei Pan, Ting Yao, Yehao Li, Yu Wang, Chong-Walt Wah Ngo, and Tao Mei. Transferrable proto- typical networks for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 0239-22247 2019.
Zhongyi Pei, Zhangjie Cao, Mingsheng Long, and Jianmin Wang. Multi-adverrarial domain adapta- tion. In Proceedings Thirty-Second AAAI Conference on Artificial Intelligence, pp. 3934-3941, 2018.
Minlong Peng, Qi Zhang. Yu-gang Jiang. and Xuan-Jing Huang. Cross-domain sentiment classifica- tion with target domain specific information. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 2505-2513, 2018.
A. Redko, Land Habrard and M. Sebban. Theoretical analysis of domain adaptation with optimal transport. In Joint European Conference Machine Learning and Knowledge Discovery in Databases, pp. 737-753. Springer, 2017.
Sam T Roweis. Em algorithms for pea and spea. In Advances in neural information processing systems, pp. 626-632, 1998.
Sebastian Ruder and Barbara Plank. Strong baselines for neural semi-supervised learning under domain shift. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 1044-1054, 2018.
K. Saito, Y. Ushiku, and T. Harada. Asymmetric tri-training for unsupervised domain adaptation. In ICML, 2018
Prathusha Kameswara Sarma, Yingyu Liang, and William Sethares. Shallow domain adaptive embeddings for sentiment analysis. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 5552-5561, 2019.
Jian Shen, Yanru Qu, Weinan Zhang, and Yong Yu. Wasserstein distance guided representation learning for domain adaptation. In AAAL, 2018.
Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In European conference computer vision, pp. 443-450 Springer, 2016.
10

---

Under review as a conference paper at ICLR 2021
Tatiana Tommasi and Barbara Caputo. Frustratingly easy nbnn domain adaptation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 897-904, 2013.
Longhui Wei, Shiliang Zhang, Wen Gao, and Qi Tian. Person transfer gan to bridge domain gap for person re-identification. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 79-88, 2018.
Fangzhao Wu and Yongfeng Huang. Sentiment domain adaptation with multiple sources. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 301-310, 2016.
Dongho Xi, Fuzhen Zhuang. Ganbin Zhou, Xiaohu Cheng, Fen Lin, and Qing He. Domain adaptation with category attention network for deep sentiment analysis. In Proceedings of The Web Conference 2020, pp. 3133 1139-3139, 2020.
Werner Zellinger. Thomas Grubinger, Edwin Lughofer, Thomas Natschläger, and Susanne Saminger- Platz. Central moment discrepancy (emd) for domain- invariant representation learning, arXiv preprint / 088/ 1, 2017.
Yftah Ziser and Roi Reichart. Pivot based language modeling for improved neural domain adaptation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 1241- 1251, 2018.
Yftah Ziser and Roi Reichart. Task refinement learning for improved accuracy and stability of unsupervised domain adaptation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 5895-5906, 2019.

---

Under review as a conference paper at ICLR 2021
A APPENDIX
A.1 SLICED WASSERSTEIN DISTANCE
We relied on minimizing the Sliced Wasserstein (SWD) distance for domain alignment. SWD is defined based on the Wasserstein distance (WD) and is mean to come with more computationally efficient distribution metric. The WD between two distributions ps and pr. defined as:
/ J c(x, w)dr(a.))
W Wo(ps, (T) = infeer(s.
(
where 1'(ps. PT) is the set of all joint distributions PS,T with marginal single variable distributions ps and pT. and c: X x Y + Rt is the cost function, e.g., f--norm Euclidean distance.
We observe that computing WD involves solving complicated optimization problem in the general case. However, when the two distributions are 1- dimienniinnll, WD has a closed-form solution:
We(ps. (LPps.PT) = I' c(P' (T), PT (T))dr,
where Ps and PT are the cumulative distributions of the ID distributions ps and pr. This closed- form solution motivates the definition of SWD in order extend applicability of equation 9 for higher dimensional distributions.
SWD is defined based on the idea of slice sampling Neal (2003) The idea is project two d- dimensional distributions into their marginal one-dimensional distributions along a subspace, i.e., slicing the dith-iinnniionl distributions, and then compute the distance between the two distribution by integrating over all the WD between the resulting ID marginal probability distributions over all possible ID subspaces using the closed form solution of WD. This can be a good replacement for the WD as any probability distribution can be expressed by the set of 1 --dimensional marginal projection distributions Helgaaoo (201T). More specifically, a one-dimensional slice of the distribution for the distribution ps is defined:
Rps(t:7)
ps(x))(( (r,, x))dx, SS--
(10)
where 5(-) denotes the Kronecker delta function, ( denotes the vector inner dot product, Sd-1 is the d-dimensional unit sphere, and y is the projection direction.
The SWD is defined the integral of all WD between the sliced distributions over all ID subspaces y on the unit sphere as follows:
SW SW(s.PPT) (T) W(Rps(s1), Rpp(s
(11)
The main advantage of using SWD is that, computing SWD does not require solving numerically expensive optimization.
In our practical setting, only samples from the distributions are available and don't have the distributional form. Another advantage of SWD is that its empirical version can be computed based on the dieeemmnnniioal empirical WD. One-dimensional empirical WD be approximated as the é,- distance between the sorted samples. We can compute merely the integrand function in equation [T] for known 7 and then the integral in equation [ipaation[[[ii via Monte Carlo style numerical integration. To this end, we draw random projection subspace y from a uniform distribution that is defined over the unit sphere and then compute ID WD along this sample. We can then approximate the integral in equation [1 by computing the arithmetic average over a suitably large enough number of drawn samples. More specifically, the SWD between f-dimensional samples (Ø(x;)) E R ps M and (d(x?) IR PT) in our setting can be approximated the following sum:
M
sws PT)
Kn, ((xx - (1u, o6xx (xx
(12)
where 7 E gf-1 is uniformly drawn random sample from the unit f f-dimensional ball g/-1, and si and [i] are the sorted indices of {7 ((xx)) for source and target domains, respectively.
12

---

Under review as a conference paper at ICLR 2021
We utilize this empirical version of SWD in equation 12 align the distributions in the embedding space. Note that the function equation 12 is differentiable with respect to the encoder parameters and hence parameters.
gradient-based optimization techniques to minimize it with respect to the model
A.2 PROOF OF THROREM 1
We use the following theorem by Redko et al. Redko & Sebban (2017) and result by Bolley Bolley al. (007) convergence of the empirical distribution the true distribution in terms of the WI) distance in our proof.
Theorem 2 (Redko et al. Redko & Sebban (2017): Under the assumptions described in framework, assume that model is trained the source domain, then for any d' d and 22, there exists a constant number No depending on such that for any & > and min [, M) 2
max(ş- (d' )2))) with probability at least 1 - E, the following holds:
er Ses + W (pT,, As) + est (2 1 )
(13)
Theorem 2 provides an upperbound for the performance of source-trained model in the target domain Redko et al. Redko & Sebban (017)) prove Theorem for a binary classification setting. We also provide our proof in this case but it can be extended.
The second term in Eq. equation 13 demonstrates the effect of domain shift on the performance of a source-trained model significant, this term will be large and hence the upperbound in Eq. equation [3 will be loose which means potential performance degradation. Our algorithm mitigates domain gap because this term minimized by minimization of the second and the third terms in Theorem 1.
target domain. When the distance between the two distributions
Theorem Consider that we the procedure described in Algorithm | for cross-domain sentiment analysis, then the following inequality holds for the target expected error:
et Ses + D(As, (P) + D(AT, (p) (1 (1- es.p + (2log( (008(2)/10
+
+2 N
M
(14)
where & is a constant which depends on C()) and (w=" denotes the expected risk of the optimally joint trained model when used on both the source domain and the pseudo-dataset,
Proof:
Due to the construction of the aseudo-datased the probability that the predicted labels for the pseudo-data points to be false equal
T. Let:
0,
yf = 8 otherwise.
C( (2.), yt)
C(h (2 o.
(15)
We Jensen's inequality and take expectation both sides of equation 15 deduce:
cpp- - cTl < (h (2?). - L(hun (z.). s (1-
(16)
Applying equation 16 in the below, deduce:
es + eT es + eT + ep ep Ses + let epl s es + ep + (1 (1-r1 -7)
(17)
Taking infimum on both sides of equation 17 we deduce:
ES. S ES.P (1 (1-7) 7).
(18)
Now by considering Theorem 2 for the two domains S and T and then using equation 18 in
equation [3] we can conclude:
eT Ses + D(JT, (s) cs, + + (2 )/5) (V N M
(
(19)
13

---

Under review as a conference paper at ICLR 2021
Now using the triangular inequality the metrics can deduce:
Dipr As) I ppp + D(us,up) D(pT, + D(ss, + (D(pp, pp). (20)
Now we replace the term D(pp, Hop) with its empirical counterpart using Theorem 1.1 in the work by Bolley et al. (0007).
Theorem 3 (Theorem 1.1 by Bolley et al. Bolley et al. (2007) )) consider that p(-) E P(Z) and Sa exp [a|| ddp()) oo for some a > 0. Let p(x) x Σ, 0(x,) denote the empirical distribution that built from the samples {m that are drawn i.d from æ; ~ p(x). Then for any
d' > d and & < 22, there exists No such that for any > 0 and N 2 No max(1, (1, e +2),, we have:
(v. > < exp( - No²). 2
N
P(W(D,))
(21)
where W denotes the WD distance. This relation measures the distance between the empirical distribution and the true distribution, expressed in the WD distance.
Applying equation 20 and equation 21 on equation 19 concludes Theorem 2 as stated:
er Ses + D(As; (bp) + D(AT, + (1 7) + es + (2log( (108(29/// (/ T + M V
(22)
A.3 COMPLEXITY ANALYSIS FOR GMM ESTIMATION
Estimating a GMM distribution usually is a computationally expensive tasks. The major reason is that normally the data points are unlabeled. This would necessitate relying iterative algorithms such expectation maximization (EM) algorithm Moon (196).. Preforming iterative E and steps until convergence leads to high computational complexity Roweis (1998). However, estimating the prototypical distribution with a GMM distribution is much simpler in our learning setting. Existence of labels helps us decouple the Gaussian components and compute the parameters using MAP estimate for each of the mode parameters in one step as follows:
S ā; N
Σ
Σ
Σ (Ess
S (dx( - (is)) (0x((=)))) (.))
S
(ar EESS
(23)
Given the above and considering that the source domain data is balanced, complexity of computing aj is O(N) (just checking whether data points x; belong to Sý). Complexity of computing μe, is O(NF/k), where F the dimension the embedding space. Complexity of computing the co- variance matrices Σj is ((F( ))). Since, we have components, the total complexity of computing GMM is O EN ). If OFF) O(k), which seems to be a reasonable practical assumption, then the total complexity of computing GMM would be O Given the large number of learnable parameters in most deep neural networks which are more than N for most cases, this complexity is fully dominated by complexity of single step of backpropagation. Hence, this computing the GMM parameters does not increase the computational complexity for.
A.4 ADDITIONAL. EXPERIMENTS ON IMBALANCED DATA
We have used balanced dataset in terms of class labels in our experiments. However, the label distribution for the target domain training dataset cannot be enforced be balanced in practical applications due to absence of labels. This a highly unexplored challenge in the domain adaptation literature which we study in this section. To study the effect of label imbalance using a controlled experiment, we synthetically design imbalanced dataset using the Amazon dataset. We repeat the experiments we had, only with the difference of using imbalanced target domain datasets. We design two experiments by only include 111 and 250 data points from class such that the target domain datasets has the 90/10 and 80/20 ratios of imbalance between the two classes, respectively. We have provided domain adaptation results using for these two imbalanced scenarios in Table We can
14

---

Under review as a conference paper at ICLR 2021
Task
B-DD
B B-+B
B-*K
D-+B
D-
D-*
80/20
82.8 = 0.3
83.2 0.5
85.5 = 0.3
78.7 = 0.2
83.3 + 0.2
86.8 + 0.2
90/10
82.9 =
83.4 + 0.3
85.8
78.5 = 0.4
83.3 + 0.4
86.8 0.3
Task
E-+B
B-D
E-K
K-B
K-D
K-E
80/20
78.7 + 0.2
78.5 0.5
88.6 + 0.1
76.3 = 0.6
77.9 0.4
86.6 + 0.1
90/10
78.7 +: 0.2
78.0 + 0.4
88.0 + 0.2
76.5 = 0.5
77.3 + 0.3
86.7 + 0.2
Table 3: Effect of label imbalance in the target domain on the proposed method.
(a) Source
(h) Proto. Dist. Samples
(c) Target: Pre-Adapt.
(d) Target: Post-Adapt.
Figure 4: UMAP visualization for the task D-*K task in the imbalanced regime of 90/ 10: (a) the source domain testing split. (b) the prototypical distribution samples, (c) the target domain testing split prior to adaptation, and (d) the target domain testing split after adaptation. (Best viewed in color).
that performance of our algorithm slightly has degraded but our algorithm has been robust to large extent with respect to label imbalance.
As secondary sanity check, we have presented the UMAP visualization for the testing data split of
the task DD>K for the imbalanced 90/10 task in Figure 4 Observations in Figure 4 match what we reported in Table 3 confirming that our algorithm does suffer considerable degradation if the target domain data imbalanced for the tasks built using the Amazon reviews dataset.
15


Reviews:
[{"review_id": "xZ_9kj54v_u", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors propose a new method to improve cross-domain sentiment analysis. They introduce large margins between classes in the source domain and it helps to reduce the possibility of misclassification on the target domain because of domain drift. After the training on the source domain, they use a Gaussian mixture model to generalize the samples and then select a subset of the drawn samples from the learned GMM for which the confidence level of the classifier is more than a predefined threshold. Therefore the drawn samples will keep larger margin between classes in the source domain. The authors provide a theoretic analysis to justify the algorithm. The experimental results demonstrate the efficacy of their method. \n\n\nPros:\n1. The authors enlarge the margin between classes on the source domain to reduce the possibility of missclassification on the target domain because of domain drift.\n2. The paper analyzes their method in a theoretic way.\n\nCons:\n1. The proposed method is very straightforward. The 'max-margin' component in the paper is implemented by the selection of samples with higher confidence of the classifier. Therefore, the performance depends on careful tuning of the threshold. A higher threshold will result in lots of samples in the source domain will be filtered out and then there is information loss during the alignment between source domain and target domain.\n\n2. The evaluation is weak. Some experimental settings are unreasonable.\n\n    a) The baseline methods use different dimensions of feature vectors. Some use 5000 and the other use 30000 as the vector size. For the fair comparison, both settings of 5000 and 30000 should be considered for the baseline methods. Perhaps some methods are sensitive to the variance of the vector size.\n\n    b) How to set the threshold \\tau? The author uses 0.95 without any explanation.\n\n    c) It is not enough that only two values, i.e. 0 and 0.99 are assigned in Table2. A line should be drawn to depict the variance of performance according to different values of the threshold.\n\nTypos:\nThere are some typos in the paper. The authors should proofread the draft carefully. I list some typos as follows:\n\n'costumers' --> 'customers'\n\n'domain shit' --> 'domain shift'\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "uu-2cccsQY_", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new domain adaptation SAUM method that learns a large margin classifier between different classes for cross-domain sentiment analysis. The key idea is to train a domain-agnostic embedding space based on learning a prototypical source distribution with GMM, which is then used to align domain distributions via SWD minimization. Experimental results on Amazon multi-domain review dataset empirically demonstrate that the proposed SAUM method can outperform the state-of-the-art domain adaptation baselines. Moreover, detailed theoretical and empirical analysis are provided to demonstrate the effectiveness of the proposed method.\n\nStrengths\n\n1.The paper is reasonably well-written and structured. \n\n2.The proposed SAUM method is technically sound.\n\n3.The proposed SAUM method can outperform state-of-the-art domain adaptation methods.\n\n4.Theoretical analysis and empirical analysis and sufficient and convinced.\n\nWeaknesses\n\n1.For domain adaptation in the NLP field, powerful pre-trained language models, e.g., BERT, XLNet, can overcome the domain-shift problem to some extent. Thus, the authors should be used as the base encoder for all methods and then compare the efficacy of the transfer parts instead of the simplest n-gram features.\n\n2.The whole procedure is slightly complex. The author formulates the prototypical distribution as a GMM, which has high algorithm complexity. However, formal complexity analysis is absent. The author should provide an analysis of the time complexity and training time of the proposed SAUM method compared with other baselines. Besides, a statistically significant test is absent for performance improvements.\n\n3.The motivation of learning a large margin between different classes is exactly discriminative learning, which is not novel when combined with domain adaptation methods and already proposed in the existing literature, e.g.,\nUnified Deep Supervised Domain Adaptation and Generalization, Saeid et al., ICCV 2017. \nContrastive Adaptation Network for Unsupervised Domain Adaptation, Kang et al.,  CVPR 2019\nJoint Domain Alignment and Discriminative Feature Learning for Unsupervised Deep Domain Adaptation, Chen et al., AAAI 2019.\n\nHowever, this paper lacks detailed discussions and comparisons with existing discriminative feature learning methods for domain adaptation.\n\n4.The unlabeled data (2000) from the preprocessed Amazon review dataset (Blitzer version) is perfectly balanced, which is impractical in real-world applications. Since we cannot control the label distribution of unlabeled data during training, the author should also use a more convinced setting as did in\nAdaptive Semi-supervised Learning for Cross-domain Sentiment Classification, He et al., EMNLP 2018, which directly samples unlabeled data from millions of reviews.\n\n5.The paper lacks some related work about cross-domain sentiment analysis, e.g., \nEnd-to-end adversarial memory network for cross-domain sentiment classification, Li et al., IJCAI 2017\nAdaptive Semi-supervised Learning for Cross-domain Sentiment Classification, He et al., EMNLP 2018\nHierarchical attention transfer network for cross-domain sentiment classification, Li et al., AAAI 18\n\n\nQuestions:\n\n1.Have the authors conducted the significance tests for the improvements?\n\n2.How fast does this algorithm run or train compared with other baselines?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "I_4nJU5F_Bw", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe paper proposes an unsupervised domain adaptation approach for sentiment analysis. The main idea is to align the cross-domain representation with Sliced Wasserstein Distance (SWD), and train the sentiment classifier on additional pseudo-labeled target domain data. By modeling the embedding spaces as a mixture of Gaussians (GMM), with one Gaussian per sentiment class, the authors sample for pseudo-labeled data with the GMM and retain only the samples that are labeled confidently by the classifier. The authors presented a theoretical analysis leveraging on previous results for SWD for domain adaptation.  \n\nStrengths:\n- the paper presents an appealing representation learning approach to cross-domain alignment of representations. \n\nWeaknesses:\n- By only retaining samples with high classifier confidence, the authors claim that \"the margin between the clusters in the source domain increase if we use the generated pseudo-dataset for domain alignment\". This seems to be an important claim in this paper, as the authors named their approach a \"max-margin\" approach, and they argue that it is this effect that is the reason for the good performance of their model. However, there is no proof in the paper of this claim, and I am not sure if that is true in a mathematical sense.\n\n- As a general domain adaptation paper, the results would be more convincing with another data set. As an application paper for cross-domain sentiment analysis, the authors failed to compare against recent work such as [2], [3] and [4]. Du et al. [2] used the BERT representation, but in Table 1 in [2], the results of IATN [3] and HATN [4] seemed to outperform this current submission without using BERT.\n\nMinor comments:\n- Wrong reference for TAT, should be [1]\n- Color in Figure 2 could be more distinct.\n- K->E: PBLM has a higher score (87.1) but SAUM^2's 86.8 was bolded\n- The paper did not explain how the standard deviations in Table 1 are obtained, for rows that have them.\n- The authors should proof read the paper for typos and grammatical mistakes, e.g., \"costumers\", \"Due to existence\" missing \"the\", \"we use the common of reducing\", \"our goal is mitigate\" missing \"to\", \"classier\", \n\n\n[1] Hong Liu, Mingsheng Long, Jianmin Wang, and Michael Jordan. Transferable adversarial training: A general approach to adapting deep classifiers. ICML 2019.\n[2] Chunning Du, Haifeng Sun, Jingyu Wang, Qi Qi, Jianxin Liao. Adversarial and Domain-Aware BERT for Cross-Domain Sentiment Analysis. ACL 2020.\n[3] Kai Zhang, Hefu Zhang, Qi Liu, Hongke Zhao, Hengshu Zhu, Enhong Chen. Interactive Attention Transfer Network for Cross-Domain Sentiment Classiﬁcation. AAAI 2019\n[4] Zheng Li, Ying Wei, Yu Zhang, Qiang Yang. Hierarchical Attention Transfer Network for Cross-Domain Sentiment Classiﬁcation. AAAI 2018\n\nComments after authors' rebuttal:\nThanks for addressing my comments. However, I think the current submission needs further work. \n- The authors agreed with me on my point that the \"max-margin\" claim might need further work, and since this is an important claim in the paper, I cannot improve my review score after the author response.\n- Different data splits should not be a barrier for comparison against previous work.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "HWLx9nMTsjd", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Pros: \nThe proposed SAUM2 is novel. As a contribution, the work introduced large margins between classes in a source domain. This is relevant in a cross-domain sentiment analysis problem. The result shows that the resultant model is domain-independent which is Ok for a general application. \n\nCons: \nAlthough the SAUM2 is novel in my opinion, the algorithm seems cumbersome. It is unclear how the \"domain shift\" would cause performance degradation. This is so because it was not shown how Theorem 1 was an explanation of Algorithm 1. The idea looks good in my assessment but it was not shown how it would be reproducible in subsequent works. \n\nThe following minor issues need to be revised: \n1. Abstract: The 3rd sentence needs to be revised - 'per each domain' is ambiguous. \n2. Introduction: Line 9 - 'in a other different domains' needs to be revised. In the second paragraph, the term 'domain-agnostic' needs to be defined or explained. On page 2, 'SWD' needs to be defined also or refer to its definition in Pg4, Para 1. Also on Page 2, under contributions, the phrase 'in the embedding matches this distribution' is ambiguous. \n3. Page 6, Para. 1: The section of the Appendix referred to should be specified. \n4. Page 8, Section 6.3: 'we may not matching the higher order' is ambiguous. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this paper, the authors proposed a large-margin-based domain adaptation method for cross-domain sentiment analysis.

The idea of developing a large-margin-based method for domain adaptation is not new. Though the proposed method contains some new ideas,  the difference between the proposed method and the existing large-margin based methods needs to be discussed and studied empirically.  In addition, the experimental results are not convincing: some related baselines are missing and experiments need to be conducted on more datasets. 

Though the authors did provide long responses to each reviewer, after a lot of discussions, the reviewers still find that their concerns are not well addressed. 

Therefore, this paper is not ready to be published in ICLR based on its current shape.

Decision: Reject