SlowGuess/ACL_2020_OCR huggingface dataset:
Deepanway Ghosal†, Devamanyu HazarikaΦ, Abhinaba RoyΦ, Navonil Majumder†, Rada Mihalceaⁿ, Soujanya Poria†
†Singapore University of Technology and Design, Singapore
$^{\Phi}$ National University of Singapore, Singapore
$^{\text{©}}$ Nanyang Technological University, Singapore
$^{\mathrm{a}}$ University of Michigan, USA
{deepanway_ghosal@mymail., sporia@, nmder@}sutd.edu.sg, {hazarika@comp.nus, abhinaba.roy@ntu}.edu.sg,mihalcea@umich.edu
Cross-domain sentiment analysis has received significant attention in recent years, prompted by the need to combat the domain gap between different applications that make use of sentiment analysis. In this paper, we take a novel perspective on this task by exploring the role of external commonsense knowledge. We introduce a new framework, KinGDOM, which utilizes the ConceptNet knowledge graph to enrich the semantics of a document by providing both domain-specific and domain-general background concepts. These concepts are learned by training a graph convolutional autoencoder that leverages inter-domain concepts in a domain-invariant manner. Conditioning a popular domain-adversarial baseline method with these learned concepts helps improve its performance over state-of-the-art approaches, demonstrating the efficacy of our proposed framework.
Sentiment Analysis (SA) is a popular NLP task used in many applications (Zhang et al., 2018). Current models trained for this task, however, cannot be reliably deployed due to the distributional mismatch between the training and evaluation domains (Daumé III and Marcu, 2006). Domain adaptation, a case of transductive transfer learning, is a widely studied field of research that can be effectively used to tackle this problem (Wilson and Cook, 2018).
Research in the field of cross-domain SA has proposed diverse approaches, which include learning domain-specific sentiment words/lexicons (Sarma et al., 2018; Hamilton et al., 2016b), co-occurrence based learning (Blitzer et al., 2007a), domain-adversarial learning (Ganin et al., 2016), among
Figure 1: ConceptNet provides networks with background concepts that enhance their semantic understanding. For example, for a target sentence from electronics domain, The software came with decent screen savers, comprising domain-specific terms like screen saver or wallpaper, ConceptNet helps connecting them to general concepts like design, thus allowing a network better understand their meaning. Furthermore, inter-domain conceptual bridge can also be established to connect source and target domains (wallpaper-sketch have similar conceptual notions under the link design).
others. In this work, we adopt the domain-adversarial framework and attempt to improve it further by infusing commonsense knowledge using ConceptNet - a large-scale knowledge graph (Speer et al., 2017).
Augmenting neural models with external knowledge bases (KB) has shown benefits across a range of NLP applications (Peters et al., 2019; Li et al., 2019; IV et al., 2019; liu et al., 2019; Bi et al., 2019). Despite their popularity, efforts to incorporate KBs into the domain-adaptation framework has been sporadic (Wang et al., 2008; Xiang et al., 2010). To this end, we identify multiple advantages of using commonsense KBs for domain adaptation.
First, KBs help in grounding text to real enti
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3198-3210 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
ties, factual knowledge, and commonsense concepts. Commonsense KBs, in particular, provide a rich source of background concepts-related by commonsense links—which can enhance the semantics of a piece of text by providing both domain-specific and domain-general concepts (Yang et al., 2019; Zhong et al., 2019; Agarwal et al., 2015; Zhong et al., 2019) (see Fig. 1). For cross-domain SA, word polarities might vary among different domains. For example, heavy can be a positive feature for a truck, but a negative feature for a smartphone. It is, however, difficult to assign contextual-polarities solely from data, especially when there is no supervision (Boia et al., 2014). In this domain-specific scenario, commonsense knowledge provides a dynamic way to enhance the context and help models understand sentiment-bearing terms and opinion targets through its structural relations (Cambria et al., 2018). They also often aid in unearthing implicitly expressed sentiment (Balahur et al., 2011).
Second, domains often share relations through latent semantic concepts (Kim et al., 2017a). For example, notions of wallpaper (from electronics) and sketch (from books) can be associated via related concepts such as design (see Fig. 1). Multi-relational KBs provide a natural way to leverage such inter-domain relationships. These connections can help models understand target-specific terms by associating to known domain-general or even source-specific concepts.
Following these intuitions, we propose a two-step modular framework, KinGDOM (Knowledge-Guided Domain adaptation), which utilizes commonsense KB for domain adaptation. KinGDOM first trains a shared graph autoencoder using a graph convolution network (GCN) on ConceptNet, so as to learn: 1) inter-domain conceptual links through multiple inference steps across neighboring concepts; and 2) domain-invariant concept representations due to shared autoencoding. It then extracts document-specific sub-graph embeddings and feeds them to a popular domain-adversarial model DANN (Ganin et al., 2016). Additionally, we also train a shared autoencoder on these extracted graph embeddings to promote further domain-invariance (Glorot et al., 2011).
Our main contributions in this work are:
1. We propose KinGDOM, a domain-adversarial framework that uses an external KB (ConceptNet) for unsupervised domain adaptation. KinG
DOM learns domain-invariant features of KB concepts using a graph autoencoding strategy.
2. We demonstrate, through experiments, that KinGDOM surpasses state-of-the-art methods on the Amazon-reviews dataset (Blitzer et al., 2007b), thus validating our claim that external knowledge can aid the task of cross-domain SA.
In the remaining paper, §2 explains related works and compares KinGDOM to them; §3 presents task definition and preliminaries; §4 introduces our proposed framework, KinGDOM; §5 discusses experimental setup followed by results and extensive analyses in §6; finally, §7 concludes this paper.
Domain adaptation methods can be broadly categorized into three approaches: a) instance-selection (Jiang and Zhai, 2007; Chen et al., 2011; Cao et al., 2018), b) self-labeling (He and Zhou, 2011) and c) representation learning (Glorot et al., 2011; Chen et al., 2012; Tzeng et al., 2014). Our focus is on the third category which has emerged as a popular approach in this deep representation learning era (Ruder, 2019; Poria et al., 2020).
Domain-adversarial Training. Our work deals with domain-adversarial approaches (Kouw and Loog, 2019), where we extend DANN Ganin et al. (2016). Despite its popularity, DANN cannot model domain-specific information (e.g. indicators of tasty, delicious for kitchen domain) (Peng et al., 2018b). Rectifications include shared-private encoders that model both domain-invariant and - specific features (Li et al., 2012; Bousmalis et al., 2016a; Kim et al., 2017b; Chang et al., 2019), using adversarial and orthogonality losses (Liu et al., 2017; Li et al., 2018). Although we do not use private encoders, we posit that our model is capable of capturing domain-specificity via the sentence-specific concept graph. Also, our approach is flexible enough to be adapted to the setup of shared-private encoders.
External Knowledge. Use of external knowledge has been explored in both inductive and transductive settings (Banerjee, 2007; Deng et al., 2018). Few works have explored external knowledge in domain adaptation based on Wikipedia as auxiliary information, using co-clustering (Wang et al., 2008) and semi-supervised learning (SSL) (Xiang et al., 2010). SSL has also been explored by Alam
et al. (2018) in the Twitter domain. Although we share a similar motivation, there exist crucial differences. Primarily, we learn graph embeddings at the concept level, not across complete instances. Also, we do not classify each concept node in the graph, which renders SSL inapplicable to our setup.
Domain Adaptation on Graphs. With the advent of graph neural networks, graph-based methods have become a new trend (Ghosal et al., 2019) in diverse NLP tasks such as emotion recognition in conversations (Poria et al., 2019). Graph-based domain adaptation is categorized based on the availability of cross-domain connections. For domain-exclusive graphs, approaches include SSL with GCNs (Shen and Chung, 2019) and domain-adversarial learning (Dai et al., 2019). For cross-domain connected graphs, co-regularized training (Ni et al., 2018) and joint-embedding (Xu et al., 2017) have been explored. We also utilize GCNs to learn node representations in our cross-domain ConceptNet graph. However, rather than using explicit divergence measures or domain-adversarial losses for domain invariance, we uniquely adopt a shared-autoencoder strategy on GCNs. Such ideas have been explored in vector-based approaches (Glorot et al., 2011; Chen et al., 2012).
Sentiment Analysis. One line of work models domain-dependent word embeddings (Sarma et al., 2018; Shi et al., 2018; K Sarma et al., 2019) or domain-specific sentiment lexicons (Hamilton et al., 2016a), while others attempt to learn representations based on co-occurrences of domain-specific with domain-independent terms (Blitzer et al., 2007a; Pan et al., 2010; Sharma et al., 2018). Our work is related to approaches that address domain-specificity in the target domain (Peng et al., 2018b; Bhatt et al., 2015). Works like Liu et al. (2018) attempts to model target-specificity by mapping domain-general information to domain-specific representations by using domain descriptor vectors. In contrast, we address relating domain-specific terms by modeling their relations with the other terms in knowledge bases like ConceptNet.
Domain adaptation deals with the training of models that can perform inference reliably in multiple domains. Across domains, it is assumed that the feature and label spaces are the same but with dis
crepancies in their feature distributions. In our setup, we consider two domains: source  $\mathcal{D}_s$  and target domain  $\mathcal{D}_t$  with different marginal data distributions, i.e.,  $P_{\mathcal{D}_s}(x) \neq P_{\mathcal{D}_t}(x)$ . This scenario, also known as the covariate shift (Elsahar and Galle, 2019), is predominant in SA applications and arises primarily with shifts in topics – causing a difference in vocabulary usage and their corresponding semantic and sentiment associations.
We account for unsupervised domain adaptation, where we are provided with labeled instances from the source domain  $D_{s}^{l} = \{(x_{i},y_{i})\}_{i = 1}^{N_{s}}$  and unlabeled instances from the target domain  $D_{t}^{u} = \{(x_{i})\}_{i = 1}^{N_{t}}$ . This is a realistic setting as curating annotations for the target domain is often expensive as well as time consuming. Given this setup, our goal is to train a classifier that can achieve good classification performance on the target domain.
3.2 Domain-Adversarial Neural Network
We base our framework on the domain-adversarial neural network (DANN) proposed by Ganin et al. (2016). DANN learns a shared mapping of both source and target domain instances  $M(\mathbf{x}_{s/t})$  such that a classifier  $C$  trained for the source domain can be directly applied for the target domain. Training of  $C$  is performed using the cross-entropy loss:
$$
\mathcal {L} _ {\mathrm {c l s}} = \mathbb {E} _ {(x _ {s}, y _ {s})} \left(- \sum_ {k = 1} ^ {K} \mathbb {1} _ {[ k = y _ {s} ]} \log C (M (x _ {s}))\right),
$$
where  $K$  is the number of labels. Both the mapping function  $M$  and the classifier  $C$  are realized using neural layers with parameters  $\theta_{M}$  and  $\theta_{C}$ .
Adversarial Loss. The core idea of DANN is to reduce domain gap by learning common representations that are indistinguishable to a domain discriminator. To learn a domain-invariant mapping, DANN uses an adversarial discriminator  $D_{adv}$  with parameters  $\theta_{D}$ , whose job is to distinguish between source and target instances,  $M(x_{s})$  vs.  $M(x_{t})$ . It is trained using the cross-entropy loss:
$$
\begin{array}{l} \mathcal {L} _ {\mathrm {a d v} _ {D}} = - \mathbb {E} _ {\mathbf {x} _ {s}} \left(\log D _ {a d v} \left(M (\mathbf {x} _ {s})\right)\right) \\ - \mathbb {E} _ {\mathbf {x} _ {t}} \left(\log \left(1 - D _ {a d v} (M (\mathbf {x} _ {t}))\right)\right). \\ \end{array}
$$
The mapping function then learns domain invariance by pitting against the discriminator in a minimax optimization with loss  $\mathcal{L}_{\mathrm{adv}_M} = -\mathcal{L}_{\mathrm{adv}_D}$  (Tzeng et al., 2017). This setup forces the features to become discriminative to the main
For our case, each instance is a review document
Figure 2: Illustration of KinGDOM: Step 1 uses GCN to learn concept representations. Step 2 feeds concept features to DANN.
All domains	DVD	Books	Kitchen	Electronics
RelatedTo (580k)	RelatedTo	RelatedTo	RelatedTo	RelatedTo
HasContext (80k)	HasContext	HasContext	IsA	IsA
IsA (60k)	IsA	IsA	Synonym	Synonym
DerivedFrom (42k)	Synonym	Synonym	DerivedFrom	DerivedFrom
Synonym (40k)	DerivedFrom	DerivedFrom	HasContext	HasContext
AtLocation (14k)	AtLocation	CapableOf	AtLocation	AtLocation
UsedFor (12k)	CapableOf	AtLocation	UsedFor	UsedFor
CapableOf (11k)	UsedFor	SimilarTo	SimilarTo	SimilarTo
SimilarTo (10k)	SimilarTo	UsedFor	CapableOf	CapableOf
Etymologically (5k)	Antonym	Antonym	Antonym	Antonym
Table 1: Top-10 relations of  $\mathcal{G}^{\prime}$  based on frequency. Top relations for each domain are also mentioned.
learning task and indistinguishable across domains. The point estimates of the parameters are decided at a saddle point using the minimax objective:
$$
\theta^{*} = \operatorname *{arg  min}_{\theta_{M,C}}\max_{\theta_{D}}\left(\mathcal{L}_{\mathrm{cls}} + \lambda   \mathcal{L}_{\mathrm{adv}_{D}}\right),
$$
where  $\lambda$  is a hyper-parameter. The minimax objective is realized by reversing the gradients of  $\mathcal{L}_{adv_D}$  when back-propagating through  $M$ .
KinGDOM aims to improve the DANN approach by leveraging an external knowledge source i.e., ConceptNet. Such a knowledge base is particularly useful for domain adaptation as it contains both domain specific and domain general knowledge. Unlike traditional word embeddings and semantic knowledge graphs (e.g. WordNet), ConceptNet is unique as it contains commonsense related information. We posit that both these properties of
ConceptNet will be highly useful for domain adaptation. KinGDOM follows a two-step approach described below:
Step 1: This step deals with training a domain-aggregated sub-graph of ConceptNet. In particular, it involves: a) Creating a sub-graph of ConceptNet based on all domains (§4.1). b) Training a graph-convolutional autoencoder to learn concept embeddings (Schlichtkrull et al., 2018) (§4.2).
Step 2: After the graph autoencoder is trained, a) we extract and pool document-relevant features from the trained graph for each instance in the dataset (§4.3). b) The corresponding graph feature vector is then fed into the DANN architecture for adversarial training (Ganin et al., 2016). To further enforce domain invariance, we also introduce a shared autoencoder to reconstruct the graph features (§4.4).
4.1 Step 1a) Domain-Aggregated Commonsense Graph Construction
We construct our domain-aggregated graph from ConceptNet (Speer et al., 2017). First, we introduce the following notation: the ConceptNet graph is represented as a directed labeled graph  $\mathcal{G} = (\mathcal{V},\mathcal{E},\mathcal{R})$ , with concepts/nodes  $^2 v_i\in \mathcal{V}$  and labeled edges  $(v_{i},r_{ij},v_{j})\in \mathcal{E}$ , where  $r_{ij}\in \mathcal{R}$  is the relation type of the edge between  $v_{i}$  and  $v_{j}$ . The concepts in ConceptNet are unigram words or n-gram phrases. For instance one such triplet from ConceptNet is [baking-oven, AtLocation, kitchen].
ConceptNet has approximately 34 million edges, from which we first extract a subset of edges. From the training documents of all domains in our dataset, we first extract the set of all the unique nouns, adjectives, and adverbs. These extracted words are treated as the seeds that we use to filter ConceptNet into a sub-graph. In particular, we extract all the triplets from  $\mathcal{G}$  which are within a distance of 1 to any of those seed concepts, resulting in a sub-graph  $\mathcal{G}' = (\mathcal{V}',\mathcal{E}',\mathcal{R}')$ , with approximately  $356k$  nodes and  $900k$  edges. This sub-graph would thus contain concepts across all domains along with inter-concept links. Looking at the sub-graph  $\mathcal{G}'$  from the lens of each domain, we can observe the top-10 relations within the domain in Table 1.
4.2 Step 1b) Knowledge Graph Pre-training
To utilize  $\mathcal{G}'$  in our task, we first need to compute a representation of its nodes. We do this by training a graph autoencoder model to perform link prediction. The model takes as input an incomplete set of edges  $\hat{\mathcal{E}}'$  from  $\mathcal{E}'$  in  $\mathcal{G}'$  and then assign scores to possible edges  $(c_1, r, c_2)$ , determining how likely are these edges to be in  $\mathcal{E}'$ . Following Schlichtkrull et al. (2018), our graph autoencoder model consists of: a R-GCN entity encoder and a DistMult scoring decoder.
Encoder Module. We employ the Relational Graph Convolutional Network (R-GCN) encoder from Schlichtkrull et al. (2018) as our graph encoder network. The power of this model comes from its ability to accumulate relational evidence in multiple inference steps from the local neighborhood around a given concept. The neighborhood-based convolutional feature transformation process always ensures that distinct domains are connected
via underlying concepts and influence each other to create enriched domain-aggregated feature vectors.
Precisely, our encoder module consists of two R-GCN encoders stacked upon one another. The initial concept feature vector  $\mathbf{g}_i$  is initialized randomly and thereafter transformed into the domain-aggregated feature vector  $\mathbf{h}_i\in \mathbb{R}^d$  using the two-step graph convolution process. The transformation process is detailed below:
$$
f (\mathbf {x} _ {i}, l) = \sigma \big (\sum_ {r \in \mathcal {R}} \sum_ {j \in N _ {i} ^ {r}} \frac {1}{c _ {i , r}} W _ {r} ^ {(l)} \mathbf {x} _ {j} + W _ {0} ^ {(l)} \mathbf {x} _ {i} \big),
$$
$$
\mathbf {h} _ {i} = \mathbf {h} _ {i} ^ {(2)} = f \big (\mathbf {h} _ {i} ^ {(1)}, 2 \big) \quad ; \quad \mathbf {h} _ {i} ^ {(1)} = f \big (\mathbf {g} _ {i}   1 \big),
$$
where  $N_{i}^{r}$  denotes the neighbouring concepts of concept  $i$  under relation  $r\in \mathcal{R}$ ;  $c_{i,r}$  is a normalization constant which either can be set in advance, such that,  $c_{i,r} = |N_i^r|$ , or can be learned in a gradient-based learning setup.  $\sigma$  is an activation function such as ReLU, and  $W_{r}^{(1 / 2)}$ ,  $W_{0}^{(1 / 2)}$  are learnable parameters of the transformation.
This stack of transformations effectively accumulates the normalized sum of the local neighborhood i.e. the neighborhood information for each concept in the graph. The self-connection ensures self-dependent feature transformation.
Decoder Module. DistMult factorization (Yang et al., 2014) is used as the scoring function. For a triplet  $(c_i, r, c_j)$ , the score  $s$  is obtained as follows:
$$
s (c _ {i}, r, c _ {j}) = \sigma (\mathbf {h} _ {c _ {i}} ^ {T} R _ {r} \mathbf {h} _ {c _ {j}}),
$$
where  $\sigma$  is the logistic function;  $\mathbf{h}_{c_i}$ ,  $\mathbf{h}_{c_j} \in \mathbb{R}^d$  are the R-GCN encoded feature vectors for concepts  $c_i$ ,  $c_j$ . Each relation  $r \in \mathcal{R}$  is also associated with a diagonal matrix  $R_r \in \mathbb{R}^{d \times d}$ .
Training. We train our graph autoencoder model using negative sampling (Schlichtkrull et al., 2018). For triplets in  $\hat{\mathcal{E}}'$  (positive samples), we create an equal number of negative samples by randomly corrupting the positive triplets. The corruption is performed by randomly modifying either one of the constituting concepts or the relation, creating the overall set of samples denoted by  $\mathcal{T}$ .
The task is set as a binary classification between the positive/negative triplets, where the model is trained with the standard cross-entropy loss:
$$
\mathcal {L} _ {\mathcal {G} ^ {\prime}} = - \frac {1}{2 | \hat {\mathcal {E}} ^ {\prime} |} \sum_ {(c _ {i}, r, c _ {j}, y) \in \mathcal {T}} (y \log s (c _ {i}, r, c _ {j}) +
$$
$$
(1 - y) \log (1 - s (c _ {i}, r, c _ {j}))).
$$
2We use node, concept, and entity interchangeably
<sup>3</sup>We use the Spacy POS Tagger: https://spacy.io/
usage/linguistic-features#pos-tagging
Once we train the autoencoder graph model, it will ensure that target domain-specific concepts (crucial for KG) can possibly be explained via domain-general concepts and further via interdomain knowledge. In other words, the encoded node representations  $\mathbf{h}_i$  will capture commonsense graph information in the form of domain-specific and domain-general features and thus will be effective for the downstream task when there is a distributional shift during evaluation.
4.3 Step 2a) Commonsense Graph Feature Extraction
The trained graph autoencoder model as explained in the previous section §4.2, can be used for feature extraction. We now describe the methodology to extract the document-specific commonsense graph features for a particular document  $x$ :
1) The first step is to extract the set of all unique nouns, adjectives, and adverbs present in the document. We call this set  $\mathcal{W}$ .
2) Next, we extract a subgraph from  $\mathcal{G}'$ , where we take all triplets for which both the constituting nodes are either in  $\mathcal{W}$  or are within the vicinity of radius 1 of any of the words in  $\mathcal{W}$ . We call this graph  $\mathcal{G}_{\mathcal{W}}'$ .
3) We then make a forward pass of  $\mathcal{G}_{\mathcal{W}}^{\prime}$  through the encoder of the pre-trained graph autoencoder model. This results in feature vectors  $\mathbf{h}_j$  for all unique nodes  $j$  in  $\mathcal{G}_{\mathcal{W}}^{\prime}$ .
4) Finally, we average over the feature vectors  $\mathbf{h}_j$  for all unique nodes in  $\mathcal{G}_{\mathcal{W}}^{\prime}$ , to obtain the commonsense graph features  $\mathbf{x}_{cg}$  for document  $x$ .
We surmise that since most documents will have both domain-specific and domain-general words in  $\mathcal{W}$ ,  $\mathbf{x}_{cg}$  will inherently capture the commonsense information likely to be helpful during domain adaptation.
4.4 Step 2b) Domain-adversarial Training
We feed the commonsense graph feature  $\mathbf{x}_{cg}$  pooled from  $\mathcal{G}_{\mathcal{W}}^{\prime}$  for document  $x$  (§4.3) into the DANN architecture (see §3.2). We proceed by learning a encoder function for the graph vector  $\mathbf{z}_{grp} = M_{\theta_G}'(\mathbf{x}_{cg})$  and combine its representation with the DANN encoder  $\mathbf{z}_{dann} = M_{\theta_M}(\mathbf{x})$  to get the final feature representation  $[\mathbf{z}_{dann};\mathbf{z}_{grp}]$ , of the document  $x$ . Here,  $[a;b]$  represents concatenation.
The task classifier  $C$  and domain-discriminator  $D_{adv}$  now takes this modified representation,  $[\mathbf{z}_{dann};\mathbf{z}_{grp}]$ , as its input instead of only  $\mathbf{z}_{dann}$ . To further enforce domain-invariance into the encoded graph representation  $\mathbf{z}_{grp}$ , we consider it as a hidden code in a traditional autoencoder and consequently add a shared decoder  $D_{recon}$  (with parameters  $\theta_R$ ) with a reconstruction loss (meansquared error):
$$
\mathcal {L} _ {\mathrm {r e c o n}} \left(X _ {s}, X _ {t}\right) = \mathcal {L} _ {\mathrm {r e c o n}} \left(X _ {s}\right) + \mathcal {L} _ {\mathrm {r e c o n}} \left(X _ {t}\right),
$$