SlowGuess/ACL_2020_OCR huggingface dataset:
$$
\mathrm {s . t .} \quad \mathcal {L} _ {\mathrm {r e c o n}} = - \mathbb {E} _ {\mathbf {x} _ {\mathbf {c g}}} \left(\| D _ {r e c o n} (\mathbf {z} _ {g r p}) - \mathbf {x} _ {\mathbf {c g}} \| _ {2} ^ {2}\right).
$$
We hypothesize that if  $\theta_R$  can reconstruct graph features for both domains, then it would ensure stronger domain invariance constraints in  $\mathbf{z}_{grp}$ . The final optimization of this domain-adversarial setup is based on the minimax objective:
$$
\theta^{*} = \operatorname *{arg  min}_{\theta_{G,M,C,R}}\max_{\theta_{D}}\left(\mathcal{L}_{\mathrm{cls}} + \lambda   \mathcal{L}_{\mathrm{adv}_{D}} + \gamma   \mathcal{L}_{\mathrm{recon}}\right),
$$
where  $\lambda$  and  $\gamma$  are hyper-parameters.
We consider the Amazon-reviews benchmark dataset for domain adaptation in SA (Blitzer et al., 2007b). This corpus consists of Amazon product reviews and ranges across four domains: Books, DVDs, Electronics, and Kitchen appliances. Each review is associated with a rating denoting its sentiment polarity. Reviews with rating up to 3 stars are considered to contain negative sentiment and 4 or 5 stars as positive sentiment. The dataset follows a balanced distribution between both labels yielding  $2k$  unlabelled training instances for each domain. Testing contains  $3k - 6k$  samples for evaluation. We follow similar pre-processing as bone by Ganin et al. (2016); Ruder and Plank (2018) where each review is encoded into a 5000-dimensional tfidf weighted bag-of-words (BOW) feature vector of unigrams and bigrams.
We follow Ganin et al. (2016) in training our network. Our neural layers i.e., DANN encoder  $(M)$ , graph feature encoder  $(M')$ , graph feature recon-. structor  $(D_{recon})$ , task classifier  $(C)$  and domain discriminator  $(D_{adv})$  are implemented with 100 dimensional fully connected layers. We use a cyclic  $\lambda$  as per (Ganin et al., 2016) and  $\gamma = 1$  after validating with  $\gamma \in \{0.5,1,2\}$ .  $25\%$  dropout is used in
Figure 3: Results of DANN vs DANN+ vs KinGDOM across different target domains. Best viewed in colour.
the fully connected layers and the model is trained with Adam (Kingma and Ba, 2015) optimizer.
5.3 Baseline Methods
In this paper, to inspect the role of external commonsense knowledge and analyze the improvement in performance it brings, we intentionally use BOW features and compare them against other baseline models that also use BOW features. This issue has also been addressed by Poria et al. (2020). The flexibility of KinGDOM allows other approaches, such as mSDA, CNN, etc. to be easily incorporated in it, which we plan to analyze in the future.
We compare KinGDOM with the following unsupervised domain adaptation baseline methods: DANN (Ganin et al., 2016) is a domain-adversarial method, based on which we develop KinGDOM ( $\S 3.2$ ); DANN+ The DANN model where we use an Adam optimizer instead of the original SGD optimizer. The network architecture and the rest of the hyperparameters are kept same; Variational Fair Autoencoder (VFAE) (Louizos et al., 2015) learns latent representations independent from sensitive domain knowledge, while retaining enough task information by using a MMD-based loss; Central Moment Discrepancy (CMD) (Zellinger et al., 2017) is a regularization method which minimizes the difference between feature representations by utilizing equivalent representation of probability distributions by moment sequences; Asym (Saito et al., 2017) is the asymmetric tri-training framework that uses three neural networks asymmetrically for domain adaptation; MT-Tri (Ruder and Plank, 2018) is similar to Asym, but uses multi-task learning; Domain Separation Networks (DSN) (Bousmalis et al., 2016b) learns to extract shared and private components of each domain. As per Peng et al. (2018a), it stands as the present state-of-the-art method for unsupervised domain adaptation; Task
Refinement Learning (TRL) (Ziser and Reichart, 2019) Task Refinement Learning is an unsupervised domain adaptation framework which iteratively trains a Pivot Based Language Model to gradually increase the information exposed about each pivot; TAT (Liu et al., 2019) is the transferable adversarial training setup to generate examples which helps in modelling the domain shift. TAT adversari-ally trains classifiers to make consistent predictions over these transferable examples; CoCMD (Peng et al., 2018a) is a co-training method based on the CMD regularizer which trains a classifier on simultaneously extracted domain specific and invariant features. CoCOMD, however, is SSL-based as it uses labeled data from the target domain. Although it falls outside the regime of unsupervised domain adaptation, we report its results to provide a full picture to the reader.
As mentioned in §5.3, we reimplemented the baseline DANN model using Adam optimizer and observed that its results has been notably underreported in many of the unsupervised domain adaptation literature for sentiment analysis (see Table 2). In the original DANN implementation (Ganin et al., 2016), Stochastic Gradient Descent (SGD) was used as the optimizer. However, in DANN+, using Adam optimizer leads to substantial performance jump that outperforms many of the recent advanced domain adaptation methods - CMD (Zellinger et al., 2017), VFAE (Louizos et al., 2015), ASym (Saito et al., 2017), and MT-Tri (Ruder and Plank, 2018).
We compare the performance of KinGDOM with its base models - DANN and DANN+. As observed in Fig. 3, KinGDOM surpasses DANN+ by  $1.4\%$  which asserts the improvement in domain-invariance due to the incorporation of external com
Source ↓ Target	DANN (5k)	DANN+ (5k)	VFAE (5k)	CMD (5k)	Asym (5k)	MT-Tri (5k)	TRL* (5k)	DSN (5k)	CoCMD* (5k)	KinGDOM (5k)	DANN+ (30k)	TAT (30k)	KinGDOM (30k)
B → D	78.4	82.6	79.9	80.5	80.7	81.2	82.2	82.8	83.1	83.1	84.7	84.5	85.0
B → E	73.3	79.9	79.2	78.7	79.8	78.0	-	81.9	83.0	82.2	83.0	80.1	83.9
B → K	77.9	81.8	81.6	81.3	82.5	78.8	82.7	84.4	85.3	85.0	84.0	83.6	86.6
D → B	72.3	80.3	75.5	79.5	73.2	77.1	-	80.1	81.8	81.4	82.7	81.9	82.7
D → E	75.4	79.9	78.6	79.7	77.0	81.0	-	81.4	83.4	81.7	83.4	81.9	83.9
D → K	78.3	83.0	82.2	83.0	82.5	79.5	-	83.3	85.5	84.6	85.3	84.0	87.1
E → B	71.3	74.9	72.7	74.4	73.2	73.5	-	75.1	76.9	76.9	77.1	83.2	78.4
E → D	73.8	78.6	76.5	76.3	72.9	75.4	75.8	77.1	78.3	78.8	79.6	77.9	80.3
E → K	85.4	88.6	85.0	86.0	86.9	87.2	-	87.2	87.3	88.4	89.0	90.0	89.4
K → B	70.9	75.9	72.0	75.6	72.5	73.8	72.1	76.4	77.2	78.2	77.1	75.8	80.0
K → D	74.0	79.2	73.3	77.5	74.9	77.8	-	78.0	79.6	80.7	81.3	77.7	82.3
K → E	84.3	86.9	83.8	85.4	84.6	86.0	-	86.7	87.2	87.4	88.0	88.2	88.6
Avg.	76.3	80.9	78.4	79.8	78.4	79.1	-	81.2	82.4	82.3	82.9	82.4	84.0
Table 2: Comparison with different baseline and state-of-the-art models (\$5.3). TRL* reported results on four combinations. CoCMD* is a semi-supervised domain adaptation method. DSN is the current state-of-the-art for unsupervised domain adaptation on the Amazon reviews dataset. Scores for MT-Tri are extrapolated from the graphs illustrated in Ruder and Plank (2018). Note: B: Books, D: DVD, E:Electronics, and K: Kitchen domains. 5k, 30k signify 5000 and 30,000 dimensional BOW features.
monsense knowledge.
Next, we look at Table 2 where comparisons are made with other baselines, including the state-of-the-art DSN approach. As observed, KinGDOM outperforms DSN in all the task scenarios, indicating the efficacy of our approach. Blitzer et al. (2007b), in their original work, noted that domain transfer across the two groups of DVD, Books and Electronics, Kitchen is particularly challenging. Interestingly, in our results, we observe the highest gains when the source and target domains are from these separate groups (e.g., Kitchen  $\rightarrow$  DVD, Kitchen  $\rightarrow$  Books, Electronics  $\rightarrow$  Books).
In Table 2, we also compare KinGDOM against CoCMD and TAT. Although CoCMD is a semi-supervised method, KinGDOM surpasses its performance in several of the twelve domain-pair combinations and matches its overall result without using any labelled samples from the target domain. TAT is the state-of-the-art method for unsupervised domain adaptation in the Amazon reviews dataset when used with 30,000 Bag-Of-Words (BOW) features. Interestingly, KinGDOM used with 5000 BOW features can match TAT with 30,000 BOW features and outperforms TAT by around  $1.6\%$  overall when used with the same 30,000 BOW features. The reimplementation of DANN - DANN+ with 30,000 BOW also surpasses the result of TAT by  $0.5\%$ . The results indicate that external knowledge,
when added to a simple architecture such as DANN, can surpass sophisticated state-of-the-art models, such as DSN and TAT. Our primary intention to utilize DANN as the base model is to highlight the role of knowledge base infusion in domain adaptation, devoid of sophisticated models, and complex neural maneuvering. Nevertheless, the flexibility of KinGDOM allows it to be associated with advanced models too (e.g., DSN, TAT), which we believe could perform even better. We intend to analyze this in the future.
6.1 Ablation Studies
We further analyze our framework and challenge our design choices. Specifically, we consider three variants of our architecture based on alternative ways to condition DANN with the graph features. Each of these variants reveals important clues regarding the invariance properties and task appropriateness of  $\mathbf{z}_{grp}$ . Variant 1 denotes separate decoders  $D_{recon}$  for source and target domains. In Variant 2, domain classifier  $D_{adv}$  takes only  $\mathbf{z}_{dann}$  as input whereas the sentiment classifier  $C$  takes the concatenated feature  $[\mathbf{z}_{dann};\mathbf{z}_{grp}]$ . Finally, in Variant 3,  $D_{adv}$  takes input  $[\mathbf{z}_{dann};\mathbf{z}_{grp}]$  whereas  $C$  only takes  $\mathbf{z}_{dann}$ . As seen in Fig. 4, all the variants perform worse than KinGDOM. For Variant 1, the performance drop indicates that having a shared decoder  $D_{recon}$  in KinGDOM facilitates
Figure 4: Average accuracy  $(\%)$  on target domains across different variants defined in §6.1. Best viewed in colour.
learning invariant representations and helps target domain classification. For Variant 2, removal of  $\mathbf{z}_{grp}$  from domain classifier diminishes the domain-invariance capabilities, thus making the domain classifier stronger and leading to a drop in sentiment classification performance. For Variant 3, removal of  $\mathbf{z}_{grp}$  from sentiment classifier  $C$  degrades the performance. This indicates that in KinGDOM,  $\mathbf{z}_{grp}$  contain task appropriate features retrieved from external knowledge (see §1).
Besides ablations, we also look at alternatives to the knowledge graph and bag-of-words representation used for the documents. For the former, we consider replacing ConceptNet with WordNet (Fellbaum, 2010), which is a lexical knowledge graph with conceptual-semantic and lexical connections. We find the performance of KinGDOM with WordNet to be  $1\%$  worse than ConceptNet in terms of average accuracy score. This indicates the compatibility of ConceptNet with our framework. However, the competitive performance with WordNet also suggests the usability of our framework with any structural resource comprising inter-domain connections. For the latter, we use Glove-averaged embeddings with DANN. Glove is a popular word embedding method which captures semantics using co-occurrence statistics (Pennington et al., 2014). Results in Fig. 4 show that using only Glove does not provide the amount of conceptual semantics available in ConceptNet.
6.2 Case Studies
We delve further into our results and qualitatively analyze KinGDOM. We look at a particular test document from DVD domain, for which KinGDOM predicts the correct sentiment, both when the
Figure 5: Domain-general term graphic bridges the commonsense knowledge between domain-specific terms in Electronics, Books and DVD.
source domain is Electronics and also Books. In similar settings, DANN mispredicts the same document. Looking at the corresponding document-specific sub-graph for this document, we observe conceptual links to both domain-general concepts and domain-specific concepts from the source domain. In Fig. 5, we can see the domain-specific terms CGI and film to be related to the general concept graphic which is further linked to domain-specific concepts like graphics card, writing, etc. from Electronics, Books, respectively. This example shows how KinGDOM might use these additional concepts to enhance the semantics as required for sentiment prediction.
In this paper, we explored the role of external commonsense knowledge for domain adaptation. We introduced a domain-adversarial framework called KinGDOM, which relies on an external commonsense KB (ConceptNet) to perform unsupervised domain adaptation. We showed that we can learn domain-invariant features for the concepts in the KB by using a graph convolutional autoencoder. Using the standard Amazon benchmark for domain adaption in sentiment analysis, we showed that our framework exceeds the performance of previously proposed methods for the same task. Our experiments demonstrate the usefulness of external knowledge for the task of cross-domain sentiment analysis. Our code is publicly available at https://github.com/declare-lab/kingdom.
This research is supported by A\*STAR under its RIE 2020 Advanced Manufacturing and Engineering (AME) programmatic grant, Award No. - A19E2b0098.
Basant Agarwal, Namita Mittal, Pooja Bansal, and Sonal Garg. 2015. Sentiment analysis using common-sense and context information. Comp. Int. and Neurosci., 2015:715730:1-715730:9.
Firoj Alam, Shafiq R. Joty, and Muhammad Imran. 2018. Domain adaptation with adversarial training and graph embeddings. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pages 1077-1087.
Alexandra Balahur, Jesús M. Hermida, and Andrés Montoyo. 2011. Detecting implicit expressions of sentiment in text based on commonsense knowledge. In Proceedings of the 2nd Workshop on Computational Approaches to Subjectivity and Sentiment Analysis, WASSA@ACL 2011, Portland, OR, USA, June 24, 2011, pages 53-60. Association for Computational Linguistics.
Somnath Banerjee. 2007. Boosting inductive transfer for text classification using wikipedia. In *The Sixth International Conference on Machine Learning and Applications*, ICMLA 2007, Cincinnati, Ohio, USA, 13-15 December 2007, pages 148-153. IEEE Computer Society.
Himanshu Sharad Bhatt, Deepali Semwal, and Shourya Roy. 2015. An iterative similarity based adaptation technique for cross-domain text classification. In Proceedings of the 19th Conference on Computational Natural Language Learning, CoNLL 2015, Beijing, China, July 30-31, 2015, pages 52-61.
Bin Bi, Chen Wu, Ming Yan, Wei Wang, Jiangnan Xia, and Chenliang Li. 2019. Incorporating external knowledge into machine reading for generative question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2521-2530, Hong Kong, China. Association for Computational Linguistics.
John Blitzer, Mark Dredze, and Fernando Pereira. 2007a. Biographies, bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. In ACL 2007, Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics, June 23-30, 2007, Prague, Czech Republic.
John Blitzer, Mark Dredze, and Fernando Pereira. 2007b. Biographies, Bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 440-447, Prague, Czech Republic. Association for Computational Linguistics.
Marina Boia, Claudiu Cristian Musat, and Boi Faltings. 2014. Acquiring commonsense knowledge for
sentiment analysis through human computation. In Proceedings of the Twenty-Eighth AAAI Conference on Artificial Intelligence, July 27-31, 2014, Quebec City, Quebec, Canada, pages 901-907. AAAI Press.
Konstantinos Bousmalis, George Trigeorgis, Nathan Silberman, Dilip Krishnan, and Dumitru Erhan. 2016a. Domain separation networks. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 343-351.
Konstantinos Bousmalis, George Trigeorgis, Nathan Silberman, Dilip Krishnan, and Dumitru Erhan. 2016b. Domain separation networks. In Advances in neural information processing systems, pages 343-351.
Erik Cambria, Soujanya Poria, Devamanyu Hazarika, and Kenneth Kwok. 2018. Senticnet 5: Discovering conceptual primitives for sentiment analysis by means of context embeddings. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 1795-1802. AAAI Press.
Zhangjie Cao, Mingsheng Long, Jianmin Wang, and Michael I. Jordan. 2018. Partial transfer learning with selective adversarial networks. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 2724-2732.
Wei-Lun Chang, Hui-Po Wang, Wen-Hsiao Peng, and Wei-Chen Chiu. 2019. All about structure: Adapting structural information across domains for boosting semantic segmentation. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 1900-1909.
Minmin Chen, Kilian Q. Weinberger, and John Blitzer. 2011. Co-training for domain adaptation. In Advances in Neural Information Processing Systems 24: 25th Annual Conference on Neural Information Processing Systems 2011. Proceedings of a meeting held 12-14 December 2011, Granada, Spain, pages 2456-2464.
Minmin Chen, Zhixiang Eddie Xu, Kilian Q. Weinberger, and Fei Sha. 2012. Marginalized denoising autoencoders for domain adaptation. In Proceedings of the 29th International Conference on Machine Learning, ICML 2012, Edinburgh, Scotland, UK, June 26 - July 1, 2012.
Quanyu Dai, Xiao Shen, Xiao-Ming Wu, and Dan Wang. 2019. Network transfer learning via adversarial domain adaptation with graph convolution. CoRR, abs/1909.01541.
Hal Daumé III and Daniel Marcu. 2006. Domain adaptation for statistical classifiers. J. Artif. Intell. Res., 26:101-126.
Yang Deng, Ying Shen, Min Yang, Yaliang Li, Nan Du, Wei Fan, and Kai Lei. 2018. Knowledge as A bridge: Improving cross-domain answer selection with external knowledge. In Proceedings of the 27th International Conference on Computational Linguistics, COLING 2018, Santa Fe, New Mexico, USA, August 20-26, 2018, pages 3295-3305.
Hady Elsahar and Matthias Galle. 2019. To annotate or not? predicting performance drop under domain shift. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2163-2173, Hong Kong, China. Association for Computational Linguistics.
Christiane Fellbaum. 2010. Wordnet. In Theory and applications of ontology: computer applications, pages 231-243. Springer.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario March, and Victor Lempitsky. 2016. Domain-adversarial training of neural networks. Journal of Machine Learning Research, 17(59):1-35.
Deepanway Ghosal, Navonil Majumder, Soujanya Poria, Niyati Chhaya, and Alexander Gelbukh. 2019. Dialoguegen: A graph convolutional neural network for emotion recognition in conversation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 154-164.
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. 2011. Domain adaptation for large-scale sentiment classification: A deep learning approach. In Proceedings of the 28th International Conference on Machine Learning, ICML 2011, Bellevue, Washington, USA, June 28 - July 2, 2011, pages 513-520.
William L. Hamilton, Kevin Clark, Jure Leskovec, and Dan Jurafsky. 2016a. Inducing domain-specific sentiment lexicons from unlabeled corpora. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pages 595-605. The Association for Computational Linguistics.
William L. Hamilton, Jure Leskovec, and Dan Jurafsky. 2016b. Diachronic word embeddings reveal statistical laws of semantic change. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1489-1501, Berlin, Germany. Association for Computational Linguistics.
Yulan He and Deyu Zhou. 2011. Self-training from labeled features for sentiment analysis. Inf. Process. Manage., 47(4):606-616.
Robert L. Logan IV, Nelson F. Liu, Matthew E. Peters, Matt Gardner, and Sameer Singh. 2019. Barack's wife hillary: Using knowledge graphs for fact-aware language modeling. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28-August 2, 2019, Volume 1: Long Papers, pages 5962-5971. Association for Computational Linguistics.
Jing Jiang and ChengXiang Zhai. 2007. Instance weighting for domain adaptation in NLP. In ACL 2007, Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics, June 23-30, 2007, Prague, Czech Republic.
Prathusha K Sarma, Yingyu Liang, and William Sethares. 2019. Shallow domain adaptive embeddings for sentiment analysis. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5548-5557, Hong Kong, China. Association for Computational Linguistics.
Taeksoo Kim, Moonsu Cha, Hyunsoo Kim, Jung Kwon Lee, and Jiwon Kim. 2017a. Learning to discover cross-domain relations with generative adversarial networks. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, pages 1857-1865.
Young-Bum Kim, Karl Stratos, and Dongchan Kim. 2017b. Adversarial adaptation of synthetic or stale data. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Papers, pages 1297-1307.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Wouter Marco Kouw and Marco Loog. 2019. A review of domain adaptation without target labels. IEEE transactions on pattern analysis and machine intelligence.
Lianghao Li, Xiaoming Jin, and Mingsheng Long. 2012. Topic correlation analysis for cross-domain text classification. In Proceedings of the Twenty-Sixth AAAI Conference on Artificial Intelligence, July 22-26, 2012, Toronto, Ontario, Canada. AAAI Press.
Pengfei Li, Kezhi Mao, Xuefeng Yang, and Qi Li. 2019. Improving relation extraction with knowledge-attention. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 229-239, Hong Kong, China. Association for Computational Linguistics.