SlowGuess/ACL_2020_OCR huggingface dataset:
we set the domain embedding size to 128. We train all models for 20 epochs and report the results for the model performing best on the joint development set of the open data set collection.
In this section, we present and compare the results of all the methods introduced previously. Experiments are conducted first on the open data collection introduced in Section 4.1 in the Multi-Domain and Multi-Domain with Unknown Label setups. Following, we evaluate the performance of our model on the data used for zero-shot genre NER.
The goal of these experiments is to examine the NER performance across the three proposed experimental setups which focus on model generalizability across multiple domains. We note that the results below can not be directly compared to the state-of-the-art results on each data set, as we restrict the entity types to PER, ORG, LOC, such that these types are constant across all data sets.
5.1 Multi-Domain with Known Domain Labels
First, we compare models when assuming the domain label of each test document is known at inference time. The results are listed in Table 2.
Our proposed method - MultDomain-SP-Aux (P) - obtains the best results across the entire test collection in both micro-average (+0.43) and macro-average (+1.94) compared to all other approaches and performs best on 7 out of the 8 domains. The second best method is the PoolDomain+DomainFeat which uses the domain feature as input. Our method consistently surpasses the in-domain classifiers (InDomain) on micro-average (+1.48) and macro-average (+3.11), showing the limitations of naive modeling approaches. Although increases exist across all domains, these are most prominent in domains like TC (+5.36) that have a low density of named entities and where indomain models have access to limited amounts of data. However, the in-domain performance is better than the pooled method of training, which shows consistent drops in performance on some domains (-8.69 on WB, -6.77 on BC, -1.98 on CoNLL), where information from other domains did not benefit the model.
8481
Model	Works on Unknown Domain Labels	CoNLL	Twitter	NW	BN	MZ	BC	TC	WB	μ-Avg	M-Avg
InDomain	X	89.91	67.36	91.09	91.09	86.90	84.41	77.06	64.74	85.29	81.57
InDomain+DomainClassifier	✓	88.92	66.98	90.48	90.21	85.63	84.64	76.28	59.62	83.93	80.35
PoolDomain	✓	87.93	66.21	90.86	92.76	87.73	89.06	70.29	56.05	83.94	80.11
PoolDomain-Init	✓	31.31	15.74	63.34	67.63	47.30	63.30	33.93	57.55	47.00	47.55
PoolDomain-GradRev	✓	83.49	54.55	83.95	86.87	77.46	83.93	77.78	50.88	77.29	74.86
PoolDomain+DomainFeat	X	90.74	67.80	90.32	92.27	89.12	89.86	78.40	63.37	86.34	82.74
MultDomain-SP (P)	X	87.70	59.16	88.96	93.51	88.52	89.95	77.97	55.51	82.12	80.16
MultDomain-SP (S)	✓	87.41	57.98	88.64	93.47	88.39	89.00	55.51	54.39	81.73	80.08
MultDomain-SP-Aux (P)	X	90.21	69.15	91.09	93.64	91.38	90.67	82.42	67.44	86.77	84.68
MultDomain-SP-Aux (S)	✓	88.43	67.13	91.26	93.59	87.67	89.54	78.77	59.63	84.68	82.30
Table 2: Experimental results on the eight data sets, as well as micro  $(\mu -)$  and macro (M-) averaged across data sets. Performance is measured using micro F1 score. The rows with  $\checkmark$  indicate methods that can be applied when the domain label is not known at inference time. (S) and (P) denote if inference is done through the shared (S) or private (P) layers of the architecture. Results in bold are the best across all models, those underlined are best across methods that work with unknown domain labels.
5.2 Multi-Domain with Unknown Domain Labels
We now focus on the experimental setup where domain labels are unknown for each data point at inference time. This is akin to a setup where the user is agnostic to the data the model was trained on. As only a subset of the models can perform inference in this scenario, the results are a subset of those in Table 2.
Our model - MultDomain-SP-Aux (S) - gains the best overall performance in this setup, with 1.95 macro-average F1 increase over the next best method (InDomain+DomainClassifier). The other standard baseline for domain adaptation (PoolDomain) obtains a similar performance  $(-2.19$  compared to our method) to the in-domain approach, which shows the benefits of multi-domain adaptation.
PoolDomain-Init is performing overall poorly, which shows that the INIT transfer learning strategy that is somewhat effective for source-target domain adaptation does not work well in the multidomain setup. Our intuition is that this technique is unable to learn robust features sequentially across N domains, as it performs poorly on the initial trained domains. PoolDomain-GradRev gains relatively weak performance overall, lower than the in-domain baseline.
5.3 Zero-Shot Domain
Finally, we show the results on the experimental setup where the test data is the four 'Zero-Shot Genres', which were not used in during training. Table 3 shows the experimental results of all methods that can run inference with unknown domain
Models	Zero-Shot Genres	M-Avg
A	B	C	D
InDomain+DomainClassifier	47.16	60.04	62.00	59.50	57.17
PoolDomain	52.61	62.53	63.53	61.55	60.05
PoolDomain-Init	24.38	36.92	47.13	19.47	31.98
PoolDomain-GradRev	49.48	68.97	67.95	57.41	60.95
MultDomain-SP (S)	50.9	72.27	68.19	61.86	63.30
MultDomain-SP-Aux (S)	54.50	67.77	70.30	64.02	64.15
Table 3: Evaluation results on data from genres unseen in training.
labels, as we assume that in this setup, the end-user does not have knowledge about the domains used in training and which of these are most similar to the test point.
Results show that our proposed method obtains again the best results, with a consistent margin of 2.24 macro-average F1 improvement over the next method. Pooling all data (PoolDomain) obtains better performance than building in-domain classifiers with domain classification (InDomain+DomainClassifier) unlike in the other setups. This also shows that the zero-shot domains we used are indeed different to any of the ones in training and pooling all data manages to build a slightly more robust model than individual ones trained on less data. The in-domain models perform 5.21 F1 points lower than our approach, the largest gap in all experimental setups, highlighting the robustness of the multi-domain modeling approach. The MultDomain-SP (S) model is second best, and as this is the base for our method, we discuss its performance in the ablation study from the next section.
8482
6.1 Ablation Experiments
We first focus on understanding the impact of each component added to our proposed method over the base architecture through an ablation study. Table 4 shows results using the private layer (MultDomain-SP-Aux (P)) when each of the three components are alternatively turned off: Shared-Private Linear layer, Shared-Private CRF and the domain prediction auxiliary task.
Shared vs. Shared-Private CRF With the rest of the architecture fixed, the results show that the shared-private CRF performs close to the shared CRF when the shared linear layer is used (80.08 vs. 80.16; 82.04 vs. 82.74; all comparisons in this section are on macro-average). However, once we use a separate linear layer between the BiLSTM and each CRF, the difference between having the shared and the shared-private CRFs increases drastically (81.36 vs. 83.11; 82.30 vs. 84.68). With only this late separation, the inputs to CRF decoders are still domain-independent features, which makes it hard for the linear CRF to adapt. When the inputs are already domain-dependent, the linear CRF can better use this information in performing the joint inference of the sequence. We note that only using shared-private CRF with the base architecture is equivalent to the MultDomain-SP method (Yang et al., 2017).
Shared vs. Shared-Private Linear Projections The results show that regardless of the other parameters, adding shared and private linear layers between the BiLSTM layers and the CRF(s) is always beneficial (80.08 vs. 81.36; 80.16 vs. 83.11; 82.04 vs. 82.30; 82.74 vs. 84.68). The improvements are relatively larger when combined with shared and private CRF, as previously seen.
Multi-Task Learning of Domain Labels Finally, we compare the impact of adding the multi-task learning objective. We find that, similar to the linear layers, adding the domain prediction task is always beneficial for the model with the increase being larger if is only a shared linear layer.
We expect that the two tasks at different levels of granularity rely on shared structure in the original semantic space. The document-level domain labels can help regularize the training, providing generic information about which low-level features are valuable to entity-level recognition.
6.2 InDomain with Oracle Choice
In order to understand the limitations of the multidomain setup, we study whether the models we can build from the available data could theoretically achieve better overall performance. We use an oracle-based selection technique on the in-domain models to select, after the prediction and using the gold labels the model which performed best for each test instance, as selected using F1 score or, if there are no entities, the model with most O predictions. If multiple models are tied, we choose one at random. The oracle thus provides the counter-factly "Optimal" strategy of model selection for each test instance and represents an upper bound on strategies relying on InDomain models.
Table 5 compares the oracle strategy predictions with the InDomain+DomainClassifier and the MultDomain-SP-Aux model. The results show that even though our model improves substantially over the in-domain models, an oracle selection method would push performance much higher (+6.73 F1 on the open data). This highlights both the variability of NER models trained on different data sets and that there is potentially more room for improvements in the multi-domain setup.
6.3 InDomain Models
The Supplementary Material shows a breakdown of the domain prediction labels for three methods: domain classification, domain prediction in the proposed MultDomain-SP-Aux model and the oracle in-domain choice on gold data. The oracle strategy selects the predictions from all in-domain models. Based on this, we analyzed the performance of each individual in-domain model when tested on all domains in Table 6. We find that although the Oracle strategy uses a mix of models, any model alone is unable to generalize to other domains (67.19 vs. 84.68 best InDomain model compared to the best overall model). In the zero-shot genres, the Twitter model performs close to the MultDomain-SP-Aux model (-0.56 F1), albeit it is 24 F1 lower on the multi-domain setup. This reinforces that learning shared domain features as opposed to learning individual models helps boost performance and is more robust to different types of inputs.
7 Runtime Comparison
Finally, we compare the runtime difference across various methods listed in the experiment section to test the practical implications of using our pro
8483
Auxiliary Task	Linear	CRF	CoNLL	Twitter	NW	BN	MZ	BC	TC	WB	μ-Avg	M-Avg
X	Shared	Shared	87.41	57.98	88.64	93.47	88.39	89.00	55.51	54.39	81.73	80.08
Sh-Private	87.70	59.16	88.96	93.51	88.52	89.95	77.97	55.51	82.12	80.16
X	Sh-Private	Shared	87.65	64.45	90.88	92.82	87.92	88.75	80.60	57.81	83.77	81.36
Sh-Private	89.57	67.78	90.98	92.45	90.10	88.75	80.86	64.38	85.95	83.11
✓	Shared	Shared	89.00	67.27	91.10	93.00	89.15	89.00	78.36	59.48	85.69	82.04
Sh-Private	89.48	67.19	91.31	93.48	89.99	89.48	79.18	61.84	86.55	82.74
✓	Sh-Private	Shared	88.43	67.13	91.26	93.59	87.67	89.54	78.77	59.63	84.68	82.30
Sh-Private	90.21	69.15	91.09	93.64	91.38	90.67	82.42	67.44	86.77	84.68
Table 4: Ablation study comparing the performance (F1 score) of models trained with and without: shared-private linear projections of BiLSTM outputs, shared-private CRF heads and multi-task domain classification.
Model	Open Data	Zero-Shot
InDomain + DomainClassifier	80.35	57.17
MultDomain-SP-Aux	84.68	64.15
Oracle with InDomain	91.41	80.27
Table 5: Performance in macro-average F1 of the InDomain models with an oracle model selection strategy using gold test data compared to selected methods.
Model	Open Data	Zero-Shot
CoNLL	64.26	61.40
Twitter	60.59	63.59
NW	67.19	59.00
BN	66.08	54.82
MZ	57.52	48.62
BC	59.19	46.30
TC	47.25	37.41
WB	44.09	25.41
posed multi-domain modelling approach. In test phase, we set the batch size as 128. Table 7 shows the average time of inference time used for each model. Our proposed model architecture takes 0.15 ms (33% increase) longer for inference than InDomain or PoolDomain models, which is a result of more model parameters. However, our proposed architecture is still 0.19 ms faster than using the InDomain+DomainClassifier approach.
In addition to inference runtime, we also find that the training time is not significantly more than the combined training time of N in-domain models. The main additions are that of the shared layers and the auxiliary task to the components of the N in-domain models and is thus a constant addition in the number of parameters to the total of N in-domain models. Hence, the model would scale by a constant with respect to the number of input domains (N+1 number of components, where N is the number of domains). This should allow our pro
posed model to scale to a large number of domains.
This highlights that the proposed MultDomain-SP-Aux model is a viable option for real-world applications.
Table 6: Results of InDomain models trained on each domain independently on the open data set collection and the zero-shot genres reported in macro average of F1 for each domain.
Model	Runtime (ms)
InDomain	0.45
InDomain+DomainClassifier	0.79
PoolDomain	0.45
PoolDomain-Init	0.43
PoolDomain-GradRev	0.47
PoolDomain+DomainFeat	0.45
MultDomain-SP	0.56
MultDomain-SP-Aux	0.60
Table 7: Averaged inference time (in ms) per sentence query on Open Dataset.
Robustness of NLP models is essential to their wider adoption and usability. Existing NER approaches are widely faced with limited scalability when applied to data that spans multiple domains. This paper introduced three experimental setups that provide a framework for evaluating the robustness of NER models. These include learning from data in multiple domains and testing on all domains, when the domain label of the test point is unknown and when this does not belong to a domain seen in training. Building on past research, we proposed a new neural architecture that achieves substantial improvements of up to 5 F1 points when compared to standard methods. Future work will focus on domain adaptation at the embedding layer.
Oshin Agarwal, Yinfei Yang, Byron C. Wallace, and Ani Nenkova. 2020. Interpretability analysis for named entity recognition to understand sys
8484
tem predictions and how they can improve. ArXiv, abs/2004.04564.
Alan Akbik, Tanja Bergmann, and Roland Vollgraf. 2019. Pooled contextualized embeddings for named entity recognition. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 724-728, Minneapolis, Minnesota. Association for Computational Linguistics.
Alan Akbik, Duncan Blythe, and Roland Vollgraf. 2018. Contextual string embeddings for sequence labeling. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1638-1649, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Isabelle Augenstein, Leon Derczynski, and Kalina Bontcheva. 2017. Generalisation in named entity recognition: A quantitative analysis. Computer Speech & Language, 44:61-83.
John Blitzer, Mark Dredze, and Fernando Pereira. 2007. Biographies, Bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 440-447, Prague, Czech Republic. Association for Computational Linguistics.
John Blitzer, Ryan McDonald, and Fernando Pereira. 2006. Domain adaptation with structural correspondence learning. In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing, pages 120-128, Sydney, Australia. Association for Computational Linguistics.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135-146.
Ciprian Chelba and Alex Acero. 2004. Adaptation of maximum entropy capitalizer: Little data can help a lot. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 285-292, Barcelona, Spain. Association for Computational Linguistics.
Xilun Chen, Ahmed Hassan Awadallah, Hany Hassan, Wei Wang, and Claire Cardie. 2019. Multi-source cross-lingual model transfer: Learning what to share. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3098–3112, Florence, Italy. Association for Computational Linguistics.
Xilun Chen and Claire Cardie. 2018. Multinomial adversarial networks for multi-domain text classification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1226-1240,
New Orleans, Louisiana. Association for Computational Linguistics.
Silviu Cucerzan. 2007. Large-scale named entity disambiguation based on Wikipedia data. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 708-716, Prague, Czech Republic. Association for Computational Linguistics.
Aron Culotta and Jeffrey Sorensen. 2004. Dependency tree kernels for relation extraction. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04), pages 423-429, Barcelona, Spain.
Hal Daumé III. 2007. Frustratingly easy domain adaptation. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 256-263, Prague, Czech Republic. Association for Computational Linguistics.
Jenny Rose Finkel and Christopher D Manning. 2009. Hierarchical bayesian domain adaptation. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 602-610.
R Florian, H Hassan, A Ittycheriah, H Jing, N Kambhatla, X Luo, N Nicolov, and S Roukos. 2004. A statistical model for multilingual entity detection and tracking. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004, pages 1-8, Boston, Massachusetts, USA. Association for Computational Linguistics.
Yaroslav Ganin and Victor Lempitsky. 2014. Unsupervised domain adaptation by backpropagation. arXiv preprint arXiv:1409.7495.
Hangfeng He and Xu Sun. 2017. A unified model for cross-domain and semi-supervised named entity recognition in chinese social media. In Thirty-First AAAI Conference on Artificial Intelligence, AAAI.
Eduard Hovy, Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2006. OntoNotes: The  $90\%$  solution. In Proceedings of the Human Language Technology Conference of the NAACL, Companion Volume: Short Papers, pages 57-60, New York City, USA. Association for Computational Linguistics.
Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional LSTM-CRF models for sequence tagging. ArXiv, abs/1508.01991.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
8485
Jayant Krishnamurthy and Tom M. Mitchell. 2015. Learning a compositional semantics for Freebase with an open predicate vocabulary. Transactions of the Association for Computational Linguistics, 3:257-270.
John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of the Eighteenth International Conference on Machine Learning, ICML, pages 282-289.
Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 260-270, San Diego, California. Association for Computational Linguistics.
Ji Young Lee, Franck Dernoncourt, and Peter Szolovits. 2018. Transfer learning for named-entity recognition with neural networks. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018), Miyazaki, Japan. European Languages Resources Association (ELRA).
Shoushan Li and Chengqing Zong. 2008. Multidomain sentiment classification. In Proceedings of ACL-08: HLT, Short Papers, pages 257–260, Columbus, Ohio. Association for Computational Linguistics.
Bill Yuchen Lin and Wei Lu. 2018. Neural adaptation layers for cross-domain named entity recognition. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2012-2022, Brussels, Belgium. Association for Computational Linguistics.
Gang Luo, Xiaojiang Huang, Chin-Yew Lin, and Zaiqing Nie. 2015. Joint entity recognition and disambiguation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 879-888, Lisbon, Portugal. Association for Computational Linguistics.
Xuezhe Ma and Eduard Hovy. 2016. End-to-end sequence labeling via bi-directional LSTM-CNNs-CRF. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1064-1074, Berlin, Germany. Association for Computational Linguistics.
Yishay Mansour, Mehryar Mohri, and Afshin Rostamizadeh. 2009. Domain adaptation with multiple sources. In Advances in Neural Information Processing Systems, NeurIPS, pages 1041-1048.
David McClosky, Eugene Charniak, and Mark Johnson. 2010. Automatic domain adaptation for parsing. In Human Language Technologies: The 2010 Annual