SlowGuess/NAACL_2022_OCR huggingface dataset:
Quanyu Long, Tianze Luo, Wenya Wang and Sinno Jialin Pan
{quanyu001, tianze001, wangwy, sinnopan}@ntu.edu.sg
In this work, we study Unsupervised Domain Adaptation (UDA) in a challenging self-supervised approach. One of the difficulties is how to learn task discrimination in the absence of target labels. Unlike previous literature which directly aligns cross-domain distributions or leverages reverse gradient, we propose Domain Confused Contrastive Learning (DCCL) to bridge the source and the target domains via domain puzzles, and retain discriminative representations after adaptation. Technically, DCCL searches for a most domain-challenging direction and exquisitely crafts domain confused augmentations as positive pairs, then it contrastively encourages the model to pull representations towards the other domain, thus learning more stable and effective domain invariances. We also investigate whether contrastive learning necessarily helps with UDA when performing other data augmentations. Extensive experiments demonstrate that DCCL significantly outperforms baselines.
Pre-trained language models (Devlin et al., 2019; Liu et al., 2019; Yang et al., 2019) have yielded considerable improvements with datasets drawing from various sources. However, the lack of portability of language model to adapt to a new textual domain remains a central issue (Gururangan et al., 2020), especially when the training set and testing set do not follow the same underlying distribution (changing of topic and genres) - usually referred to as domain shift. In this paper, we focus on studying Unsupervised Domain Adaptation (UDA). UDA aims at designing adaptation algorithms that attempt to generalize well on the target domain by learning from both labeled samples from the source domain and unlabeled samples from the target domain. Studying UDA fits real-world scenarios since labeled data in the target domain is usually absent. Moreover, advances in UDA will also help out-of
Figure 1: Domain puzzles that are domain-confused and overlook domain-related information, could be regarded as lying in an intermediate domain that aims to pull source and target samples closer to each other and bridge the two domains by learning domain invariant representations.
distribution generalizations (Ramponi and Plank, 2020; Krueger et al., 2021).
Extensive algorithms have been proposed to mitigate the domain shift problem, for example, domain adversarial neural network (DANN) (Ganin et al., 2016) and distribution matching (Zhuang et al., 2015). For DANN, the training process of joint optimization is unstable, requiring extensive effort to tune the hyperparameters (Shah et al., 2018; Du et al., 2020; Karouzos et al., 2021). As for distribution matching, it is very difficult to preserve the discriminative power of the model on the target task while trying to perform instance level alignment (Saito et al., 2017; Lee et al., 2019). To this end, it is essential to develop stable and effective solutions to learn domain invariance and instance-wise matching for UDA.
Recent advances in self-supervised learning (SSL), such as contrastive learning (CL), have been proven effective at instance level by leveraging raw data to define surrogate tasks that help learn repre
Human Language Technologies, pages 2982 - 2995
sentations (Chen et al., 2020a; Khosla et al., 2020; He et al., 2020; Chen et al., 2020b). CL benefits from treating instances as classes and conducting data augmentations to generate positive instance pairs. Regarding CL for UDA, previous works mark cross-domain images with the same labels (for example, real and cartoon dogs) as the positive pairs in contrastive loss (Wang et al., 2021; Park et al., 2020). However, such methods are not applicable to NLP tasks because of the massive semantic and syntactic shifts between two cross-domain sentences. Besides, from the domain adaptation perspective, constructing cross-domain positive samples and aligning domain-agnostic pairs have received less emphasis in related literature, since previous works focus on designing label preserving text transformations, such as back-translation, synonym, dropout and their combinations (Qu et al., 2021; Gao et al., 2021).
Confronting with limitations mentioned above, we propose the concept of domain puzzles which discard domain-related information to confuse the model, making it difficult to differentiate which domain these puzzles belong to. Instead of directly seeking matched sentences across the source and target domains which is infeasible, we propose to pull the source (target) data and its corresponding domain puzzles closer to reduce the domain discrepancy, as shown in Fig. 1. A simple idea to craft domain puzzles is to mask domain-specific tokens. However, token-level operations are too discrete and non-flexible to reflect the complex semantic change of natural languages. Hence, we aim to seek better domain puzzles that retain high-confidence predictions and task-discriminative power in the representation space for each training instance.
In this paper, we propose Domain Confused Contrastive Learning (DCCL) to encourage the model to learn similar representations for the original sentence and its curated domain-confused version with contrastive loss. More specifically, we synthesize these domain puzzles by utilizing adversarial examples (Zhu et al., 2020; Jiang et al., 2020). The algorithm will search for an extreme direction that roughly points to the opposite domain and produces most domain-challenging puzzles. We encourage the model to encode original and domain-confused samples closer, gradually pulling examples to the domain decision boundary as training progresses via CL, thus learning the domain invariance. Furthermore, in order to investigate whether CL necess
sarily benefits UDA, we conduct experiments and find that constructing domain puzzles as paired positive samples is favorable for UDA, however, other data augmentation methods such as back translation (Sennrich et al., 2016; Edunov et al., 2018) do not have the same effect. The experiment results show that the proposed DCCL significantly outperforms all the baselines. We also conduct quantitative experiments to measure the domain discrepancy after adaptation demonstrating that DCCL can decrease the divergence between domains in a self-supervised way. Overall, the paper makes the following contributions:
- First, a new concept of domain puzzles is put forward. We propose to craft the domain puzzles via domain-confused adversarial attack;
- Second, we propose DCCL, which is able to pull source and target samples closer to the crafted domain puzzles. The DCCL is capable of reducing domain shift and learning domain invariance;
- Third, experiments demonstrate that the proposed DCCL surpasses baselines with a large margin. We also conduct analyzing experiments to verify the effectiveness.
Problem Setup Suppose we have access to a source dataset with  $n$  labeled data points  $\mathcal{D}^S = \{x_i, y_i\}_{1,\dots,n}$  sampled i.i.d. from the source domain, and a target dataset with  $m$  unlabeled points  $\mathcal{D}^T = \{x_j\}_{1,\dots,m}$  sampled i.i.d. from the target domain, where  $x_i, x_j$  are sequences of tokens,  $y_i$  is the class label for  $x_i$ . For in-domain training with labeled training instances, the model aims to learn a function  $f(x; \theta_f, \theta_y) : x \to C$ .  $\theta_f$  is the parameter of the deep neural network encoder (e.g., pretrained language model),  $\theta_y$  denotes parameters that compute the network's class label predictions, and  $C$  is the label set. The model is learned with the following objective:
$$
\min  _ {\theta_ {f}, \theta_ {y}} \sum_ {(x, y) \sim \mathcal {D} ^ {S}} [ \mathcal {L} (f (x; \theta_ {f}, \theta_ {y}), y) ]. \tag {1}
$$
However, for Unsupervised Domain Adaptation (UDA), the goal of the adaptation algorithm is to learn a discriminative classifier from the source
Figure 2: Overview of the framework of the proposed method DCCL.
domain, which at the same time could generalize well on the target domain by leveraging unlabeled target data and learning a mapping between source and target domains. It is generally acknowledged that the discrepancy between two datasets (domain shift) can be reduced by aligning two distributions (Ben-David et al., 2006; Ben-David et al., 2010). The methods that learn domain invariant features for domain alignment include KL divergence (Zhuang et al., 2015), Maximum Mean Discrepancy (MMD) (Gretton et al., 2012), and Domain Adversarial Neural Network (DANN) (Ganin et al., 2016) (details of DANN can be found in Appendix B). DANN suffers from a vanishing gradient problem (Shen et al., 2018), and the training process is unstable (Shah et al., 2018; Du et al., 2020). Hence, more efficient and stable algorithms are essential for UDA (Wu et al., 2019b).
2.2 Adversarial Training
Adversarial training with perturbations has been shown to significantly improve the performance of the state-of-the-art language models for many natural language understanding tasks (Madry et al., 2018; Zhu et al., 2020; Jiang et al., 2020; Pereira et al., 2021). The algorithm generally considers adversarial attacks with perturbations to word embeddings and minimizes the resultant adversarial loss around the input samples. In a single domain, adversarial training (Goodfellow et al., 2015) is an inner max, outer min adversarial problem with the objective:
$$
\min _ {\theta_ {f}, \theta_ {y}} \sum_ {(x, y) \sim \mathcal {D}} [ \max _ {\delta} \mathcal {L} (f (x + \delta ; \theta_ {f}, \theta_ {y}), y) ]. (2)
$$
With (2), the standard adversarial training can also be regularized using virtual adversarial training (Miyato et al., 2018), which encourages smoothness in the embedding space. The  $\alpha_{adv}$  controls the trade-off between the two losses, usually set to
be 1.
$$
\begin{array}{l} \min  _ {\theta_ {f}, \theta_ {y}} \sum_ {(x, y) \sim \mathcal {D}} [ \mathcal {L} (f (x; \theta_ {f}, \theta_ {y}), y) + \alpha_ {a d v} \tag {3} \\ \operatorname * {m a x} _ {\delta} \mathcal {L} (f (x + \delta ; \theta_ {f}, \theta_ {y}), f (x; \theta_ {f}, \theta_ {y})) ]. \\ \end{array}
$$
For (2)(3), the inner maximization can be solved by Projected Gradient Decent (PGD) (Madry et al., 2018) with an additional assumption that the loss function is locally linear. A following iteration can approximate the adversarial perturbation  $\delta$ :
$$
\delta_ {t + 1} = \Pi_ {\| \delta \| _ {F} \leq \epsilon} (\delta_ {t} + \eta \frac {g _ {y} ^ {a d v} (\delta_ {t})}{\| g _ {y} ^ {a d v} (\delta_ {t}) \| _ {F}}), \tag {4}
$$
$$
g _ {y} ^ {\text {a d v}} \left(\delta_ {t}\right) = \nabla_ {\delta} \mathcal {L} \left(f \left(x + \delta_ {t}; \theta_ {f}, \theta_ {y}\right), y\right), \tag {5}
$$
where  $\Pi_{\| \delta \|_F \leq \epsilon}$  performs a projection onto the  $\epsilon$ -ball. The advantages of PGD lie in that it only relies on the model itself to produce diverse adversarial samples, enabling the model to generalize better to unseen data.
In this section, we focus our discussions on the proposed Domain Confused Contrastive Learning (DCCL) under a sentiment classification scenario. The overall framework of our method is illustrated in Fig. 2. The model will take source labeled and target unlabeled sentences as input. It will then augment the input data with domain puzzles by fabricating adversarial perturbations. With the augmented data, the next step produces a hidden representation for each instance with an encoder which will be further used to produce three losses to train the entire model, namely sentiment classification loss, contrastive loss and consistency loss.
3.1 Crafting domain puzzles
For UDA, Saito et al. (2017) mentions that simply matching the distributions cannot ensure high accuracy on the target domain without the target labels.
Figure 3: Two sentences sampled from Book and Music reviews. Alternatively, we can match original sentences with its degraded masked versions.
Moreover, it may cause negative transfer, deteriorating knowledge transfer from source domain to the target domain (Wang et al., 2019). Even if the matched sentences have the same label, due to huge syntactic and semantic shift, instance-based matching strategies that align examples from different domains will introduce noises for pre-trained language models, for example, aligning source domain and target domain sentences in Fig. 3.
Alternatively, we can locate and mask domain-specific tokens which are related to sentence topics and genres. Since sentences in the green box of Fig. 3 become domain-agnostic, we refer to those domain-confused sentences (one cannot tell which domain these sentences belong to) as domain puzzles. Matching distributions between the source domain and the domain puzzles, as well as the target domain and the domain puzzles, will also make language models produce domain invariant representations.
However, the domain-specific tokens are not always evident, due to the discrete nature of natural languages, it is challenging to decide correct tokens to mask without hurting the semantics especially when the sentences are complicated<sup>1</sup>. Hence, we seek domain puzzles in the representation space and introduce adversarial perturbations, because we can rely on the model itself to produce diverse but targeted domain puzzles. Note that the purpose of adversarial attack here is not to enhance the robustness, but to construct exquisitely produced perturbations for a better domain invariance in the representation space.
To generate domain-confused augmentations, we adopt adversarial attack with perturbations for domain classification. The loss for learning a domain classifier with adversarial attack can be speci
(a) Postive sampling
(b) Negative sampling
Figure 4: Proposed adaptation sampling method
fied as follows:
$$
\mathcal {L} _ {\text {d o m a i n}} = \mathcal {L} \left(f \left(x; \theta_ {f}, \theta_ {d}\right), d\right) +
$$
$$
\alpha_ {a d v} \mathcal {L} \left(f \left(x + \delta ; \theta_ {f}, \theta_ {d}\right), f \left(x; \theta_ {f}, \theta_ {d}\right)\right), \tag {6}
$$
$$
\delta = \Pi_ {\| \delta \| _ {F} \leq \epsilon} \left(\delta_ {0} + \eta \frac {g _ {d} ^ {a d v} \left(\delta_ {0}\right)}{\| g _ {d} ^ {a d v} \left(\delta_ {0}\right) \| _ {F}}\right), \tag {7}
$$
where  $\delta_0$  is the initialized noise,  $\theta_d$  is the parameter corresponding to the computation of the domain classification, and  $d$  is the domain label. Due to additional overhead incurred during fine-tuning large pre-trained language models, the number of iterations for perturbation estimation is usually 1 (Jiang et al., 2020; Pereira et al., 2021), as shown in Eq. 7. We synthesize the perturbation  $\delta$  by searching for an extreme direction that perplexes the domain classifier most in the embedding space, and  $f(x + \delta ;\theta_f)$  is the crafted domain puzzles encoded by the language model.
3.2 Learning invariance with domain puzzles
After acquiring domain puzzles, simply applying distribution matching will sacrifice discriminative knowledge learned from the source domain (Saito et al., 2017; Lee et al., 2019), and instance-based matching will also overlook global intra-domain information. To learn sentiment-wise discriminative
$^{1}$ Masking is not our focus in this paper. More detailed implementation can be found in section 4.4. We will investigate how to extract better domain tokens in our future work.
representations in the absence of the target labels, we propose to learn domain invariance via contrastive learning (CL). In general, CL benefits from the definition of the augmented positive and negative pairs by treating instances as classes (Chen et al., 2020a; Khosla et al., 2020; He et al., 2020; Chen et al., 2020b). Furthermore, the contrastive loss encourages the positive pairs to be close to each other and negative pairs to be far apart. Specifically, maximizing the similarities between positive pairs learns an invariant instance-based representation, and minimizing the similarities between negative pairs learns a uniformly distributed representation from a global view, making instances gathered near the task decision boundary away from each other (Saunshi et al., 2019; Grill et al., 2020). This will help to enhance task discrimination of the learned model.
For positive pairs, intuitively, we hope that the model could encode the original sentence and most domain-challenging examples to be closer in the representation space, gradually pulling examples to the domain decision boundary as training progresses. For negative sampling, it widens the sentiment decision boundary and promotes better sentiment-wise discriminative features for both domains. However, for cross-domain negative sampling, the contrastive loss may push the negative samples in the target (source) domain away from the anchor in the source (target) domain (see Fig. 4 (b) left). This is contradictory to the objective of domain puzzles which try to pull different domains closer. To avoid the detriment of cross-domain repulsion, excluding samples with different domains from the negative set is of great importance. Therefore, we write the following contrastive infoNCE loss (Chen et al., 2020a) as follow:
$$
\mathcal {L} _ {\text {c o n t r a s t}} = - \frac {1}{N} \sum_ {i} ^ {N} \log \frac {\exp \left(s \left(z _ {i} , z _ {i} ^ {\prime}\right) / \tau\right)}{\sum_ {k} ^ {N} \mathbb {1} _ {k \neq i} \exp \left(s \left(z _ {i} , z _ {k}\right) / \tau\right)}, \tag {8}
$$
where  $N$  is the mini batch size with samples from the same domain,  $z_{i} = g(f(x_{i};\theta_{f}))$ , and  $g(\cdot)$  is one hidden layer projection head. We denote  $x^{\prime} = x + \delta$  as the domain puzzle augmentation,  $s(\cdot)$  computes cosine similarity,  $\mathbb{1}_{k\neq i}$  is the indicator function, and  $\tau$  is the temperature hyperparameter.
3.3 Consistency Regularization
Given perturbed embedding  $x + \delta$ , which is crafted based on domain classification, we also encourage the model to produce consistent sentiment predictions with that of the original instance  $f(x; \theta_f, \theta_y)$ .
Algorithm 1 DCCL
Input: For simplicity,  $\theta$  is the parameter of the whole model.  $T$ : the total number of iterations,  $(x,y)\sim \mathcal{D}^S$ : source dataset with sentiment label  $y$ ,  $(x,d)\sim \mathcal{D}^{S}\mathcal{D}^{D}$ : source and target dataset with domain label  $d$ ,  $K$ : the number of iterations for updating  $\delta$ ,  $\sigma^2$ : the initialized variance,  $\epsilon$ : perturbation bound,  $\eta$ : the step size,  $\gamma$ : global learning rate,  $N$ : batch size,  $\tau$ : temperature,  $g(\cdot)$ : one hidden layer projection head.  $\alpha_{adv}$ ,  $\alpha$ ,  $\lambda$  and  $\beta$ : weighting factor.
1: for epoch  $= 1$  ..,T do
2: for minibatch  $N$  do
3:  $\delta \gets \mathcal{N}(0, \sigma^2 I)$
4: for  $m = 1,..,K$  do
5:  $g_{d}^{adv}\gets \nabla_{\delta}\mathcal{L}(f(x + \delta ;\theta),d)$
6:  $\delta \gets \Pi_{\| \delta \| _F\leq \epsilon}(\delta +\eta g_d^{adv} / \| g_d^{adv}\| _F)$
8:  $\mathcal{L}_{\mathrm{domain}}\gets \mathcal{L}(f(x;\theta),d)$
$$
+ \alpha_ {a d v} \mathcal {L} (f (x + \delta ; \theta), d)
$$
9:  $z = g(f(x;\theta))$
10:  $z^{\prime} = g(f(x + \delta ;\theta))$
11: for  $i = 1,\dots,N$  and  $j = 1,\dots,N$  do
12:  $s_i' = z_i^\top z_j' / \|z_i\| \|z_j'\|$
13:  $s_{i,j} = z_i^\top z_j / \| z_i\| \| z_j\|$
14: end for
15:  $\mathcal{L}_{\mathrm{contrast}}\gets -\frac{1}{N}\sum_{i}^{N}\log \frac{\exp(s_i' / \tau)}{\sum_{j = i}^{N}\mathbb{1}_{j\neq i}\exp(s_{i,j} / \tau)}$
16:  $\mathcal{L}_{\mathrm{consist}}\gets \mathcal{L}(f(x;\theta),f(x + \delta ;\theta))$
17:  $g_{\theta}\gets \nabla_{\theta}\mathcal{L}(f(x;\theta),y) + \alpha \nabla_{\theta}\mathcal{L}_{\mathrm{domain}}$
$$
+ \lambda \nabla_ {\theta} \mathcal {L} _ {\text {c o n t r a s t}} + \beta \nabla_ {\theta} \mathcal {L} _ {\text {c o n s i s t}}
$$
18:  $\theta \gets \theta -\gamma g_{\theta}$
For this, we minimize the symmetric KL divergence, which is formulated as:
$$
\mathcal {L} _ {\text {c o n s i s t}} = \mathcal {L} \left(f \left(x; \theta_ {f}, \theta_ {y}\right), f \left(x + \delta ; \theta_ {f}, \theta_ {y}\right)\right). \tag {9}
$$
For overall training objective, we train the neural network in an end-to-end manner with a weighted sum of losses as follows.
$$
\begin{array}{l} \min  _ {\theta_ {f}, \theta_ {y}, \theta_ {d}} \sum_ {(x, y) \sim \mathcal {D} ^ {S}} \mathcal {L} (f (x; \theta_ {f}, \theta_ {y}), y) + \tag {10} \\ \sum_ {(x, y) \sim \mathcal {D} ^ {S}, \mathcal {D} ^ {T}} [ \alpha \mathcal {L} _ {\text {d o m a i n}} + \lambda \mathcal {L} _ {\text {c o n t r a s t}} + \beta \mathcal {L} _ {\text {c o n s i s t}} ]. \\ \end{array}
$$
Details of proposed DCCL are summarized in Algorithm 1.
Amazon Benchmark (Blitzer et al., 2007) $^{2}$ . We conduct experiments on this dataset for completeness since most of previous works report results on it. The dataset contains four domains: Book (BK), DVD (D), Electronics (E) and Kitchen housewares (K). There are 2,000 balanced labeled data for each domain, we randomly select 20,000 unlabeled reviews for BK, D and E. For K, only 17,856 unlabeled reviews are available.
Amazon review dataset (He et al., 2018) $^3$ . This dataset considers neutral instances which may not bias the dataset and bring more challenges $^4$ . The dataset also contains four domains: Book (BK), Electronics (E), Beauty (BT), and Music (M). Following He et al. (2018), we treat set 1 as labeled dataset containing 6,000 instances, and treat set 2 as unlabeled dataset which also contains 6,000 instances. More details about two datasets can be found in Appendix A.
For unsupervised adaptation setting, we should not have access to target labeled data at the training phase, so trained models with minimum classification error on the source validation set is saved for evaluation. At this point, we suppose a good model that generalizes well on the target domain is able to reach high performance on both validation and test set at the same time. We evaluate our model with 5 runs in all experiments, and we report the average score, standard deviation and paired t-test results.
For pre-trained language model, we use BERT base uncased (Devlin et al., 2019) as the basis for all experiments. The max length is set to 512. For optimizer, we use AdamW (Kingma and Ba, 2015; Loshchilov and Hutter, 2019) with weight decay 0.01 (for BERT baseline, we set 1e-4). We set the learning rate as 1e-5, and we use a linear scheduler with warm-up steps 0.1 of total training steps.
We set the number of adversarial iterations to be 1, adversarial weighting factor  $\alpha_{adv} = 1$  and we use  $l_{2}$  norm to compute projections. We also
follow Zhu et al. (2020) to set other adversarial hyperparameters, e.g., adversarial step size  $\eta = 5\mathrm{e} - 2$ , perturbation bound  $\epsilon = 5\mathrm{e} - 2$ . For weighting factors, we set  $\alpha = 1\mathrm{e} - 3$ ,  $\lambda = 3\mathrm{e} - 2$  and  $\beta = 5$ . We train the model with 8 epochs, temperature  $\tau$  and batch size  $N$  will be discussed later. Each adaptation requires half one hour on one A-100.