SlowGuess/ACL_2020_OCR huggingface dataset:
In this section, we present the experimental results for detecting experiment-describing sentences, entity mention extraction and experiment slot identification. For tokenization, we employ ChemDataExtractor, which is optimized for dealing with chemical formulas and unit mentions.
We tune our models in a 5-fold cross-validation setting. We also report the mean and standard deviation across those folds as development results. For the test set, we report the macro-average of the scores obtained when applying each of the five models to the test set. To put model performance in relation to human agreement, we report the corresponding statistics obtained from our inter-annotator agreement study (Section 5). Note that these numbers are based on a subset of the data and are hence not directly comparable.
Hyperparameters and training. The BiLSTM models are trained with the Adam optimizer (Kingma and Ba, 2015) with a learning rate of 1e-3. For fine-tuning the original BERT models, we follow the configuration published by Wolf et al. (2019) and use AdamW (Loshchilov and Hutter, 2019) as optimizer and a learning rate of 4e-7 for sentence classification and 1e-5 for sequence tagging. When adding BERT tokens to the BiLSTM, we also use the AdamW optimizer for the whole model and learning rates of 4e-7 or 1e-5 for the BERT part and 1e-3 for the remainder. For regularization, we employ early stopping on the development set. We use a stacked BiLSTM with two hidden layers and 500 hidden units for all tasks with the exception of the experiment sentence de
$^{5}$ https://github.com/huggingface/ transformers  
 $^{6}$ We use sklearn, https://scikit-learn.org.
<sup>7</sup>We use sklearnnn-pycrfsuite, https://pypi.org/project/sklearn-pycrfsuite.<sup>8</sup> http://chemdataextractor.org
Model	dev F1	test
P	R	F1
RBF SVM	54.2+/-3.7	64.6	54.9	59.4
Logistic Regression	53.0+/-4.2	68.2	50.9	58.3
BiLSTM mat2vec	49.9+/-3.1	49.6	69.4	57.8
BiLSTM word2vec	52.3+/-4.6	51.1	65.3	57.4
+ mat2vec	55.9+/-4.2	52.0	59.0	55.3
+bpe	58.6+/-3.0	58.9	64.7	61.7
+BERT-base	66.8+/-4.9	60.2	71.7	65.4
+SciBERT	67.9+/-4.0	58.6	74.6	65.6
BiLSTM BERT-base	64.7+/-4.6	63.7	69.9	66.7
BiLSTM SciBERT	68.1+/-3.7	60.2	73.4	66.1
BERT-base	66.0+/-4.6	58.6	71.1	64.2
SciBERT	67.9+/-4.0	60.8	74.6	67.0
BERT-large	64.3+/-4.3	63.1	75.1	68.6
humans	78.3	81.1	75.6	78.3
tection task, where we found one BiLSTM layer to work best. The attention layer of the sentence detection model has a hidden size of 100.
Experiment sentence detection. Table 5 shows our results on the detection of experiment-describing sentences. The neural models with byte-pair encoding embeddings or BERT clearly outperform the SVM and logistic regression models. Within the neural models, BERT and SciBERT add the most value, both when using their embeddings as another input to the BiLSTM and when finetuning the original BERT models. Note that even the general-domain BERT is strong enough to cope with non-standard domains. Nevertheless, models based on SciBERT outperform BERT-based models, indicating that in-domain information is indeed beneficial. For performance reasons, we use BERT-base in our experiments, but for the sake of completeness, we also run BERT-large for the task of detecting experiment sentences. Because it did not outperform BERT-base in our cross-validation based development setting, we did not further experiment with BERT-large. However, we found that it resulted in the best F1-score achieved on our test set. In general, SciBERT-based models provide very good performance and seem most robust across dev and test sets. Overall, achieving F1-scores around 67.0-68.6, such a retrieval model may already be useful in production. However, there certainly is room for improvement.
Table 5: Experiments: identifying experiment-describing sentences. P, R and F1 for experiment-describing sentences. With the exception of SVM, we downsample the non-experiment-describing sentences of the training set by 0.3.
Model	EXP.	MAT.	VAL.	DEV.	avg.
CRF	61.4	42.3	73.6	64.1	60.3
BiLSTM mat2vec	47.1	52.4	60.9	46.1	51.6
BiLSTM word2vec	55.8	58.6	59.1	51.7	56.3
+mat2vec	57.9	75.2	64.3	61.5	64.7
+bpe	63.3	81.6	68.0	68.1	70.2
+BERT-base	76.0	88.1	72.9	81.5	79.7
+SciBERT	76.9	89.8	74.1	85.2	81.5
BiLSTM BERT-base	75.4	87.6	72.6	80.8	79.1
BiLSTM SciBERT	77.1	89.9	72.1	85.7	81.2
BERT-base	81.8	70.6	88.2	73.1	78.4
SciBERT	84.5	77.0	91.6	72.7	81.5
humans	94.3	95.9	95.5	97.5	95.8
Table 6: Experiments: entity mention detection and typing. Results on test set (experiment-describing sentences only) in terms of F1, rightmost column shows the macro-average.
Entity mention extraction. Table 6 provides our results on entity mention detection and typing. Models are trained and results are reported on the subset of sentences marked as experiment-describing in the gold standard, amounting to 4,590 entity mentions in total. $^{9}$  The CRF baseline achieves comparable or better results than the BiLSTM with word2vec and/or mat2vec embeddings. However, adding subword-based embeddings (bpe and/or BERT) significantly increases performance of the BiLSTM, indicating that there are many rare words. Again, the best results are obtained when using BERT or SciBERT embeddings or when using the original SciBERT model. It is relatively easy for all model variants to recognize VALUE as these mentions usually consist of a number and unit which the model can easily memorize. Recognizing the types MATERIAL and DEVICE, in contrast, is harder and may profit from using gazetteer-based extensions.
Experiment slot filling. Table 7 shows the macro-average F1 scores for our different models on the slot identification task. $^{10}$  As for entity typing, we train and evaluate our model on the subset of sentences marked as experiment-describing, which contain 4,263 slot instances. Again, the CRF baseline outperforms the BiLSTM when using only
<sup>9</sup>The SOFC-Exp gold standard marks all entity mentions that correspond to one of the four relevant types occurring in these sentences, regardless of whether the mention fills a slot in an experiment or not.
<sup>10</sup>We evaluate on the 16 slot types as listed in Table 11. When training our model, we use the additional types experiment-evoking_word and Thickness, which are not frame slots but related annotations present in our data, see guidelines.
Model	dev	test
CRF	45.3+/-5.6	41.3
BiLSTM mat2vec	25.9+/-11.2	22.5
BiLSTM word2vec	27.5+/-9.0	27.0
+ mat2vec	43.0+/-11.5	34.9
+bpe	50.2+/-11.8	38.9
+BERT-base	64.6+/-12.8	54.2
+ SciBERT	67.1+/-13.3	59.7
BiLSTM BERT-base	63.3+/-12.9	57.4
BiLSTM SciBERT	67.8+/-12.9	62.6
BERT-base	63.4+/-13.8	54.9
SciBERT	65.6+/-13.2	56.4
humans	83.4
mat2vec and/or word2vec embeddings. The addition of BERT or SciBERT embeddings improves performance. However, on this task, the BiLSTM model with (Sci)BERT embeddings outperforms the fine-tuned original (Sci)BERT model. Compared to the other two tasks, this task requires more complex reasoning and has a larger number of possible output classes. We assume that in such a setting, adding more abstraction power to the model (in the form of a BiLSTM) leads to better results.
For a more detailed analysis, Table 8 shows the slot-wise results for the non-neural CRF baseline and the model that performs best on the development set: BiLSTM with SciBERT embeddings. As in the case of entity mention detection, the models do well for the categories that consist of numeric mentions plus particular units. In general, model performance is also tied to the frequency of the slot types in the dataset. Recognizing the role a material plays in an experiment (e.g., AnodeMaterial vs. CathodeMaterial) remains challenging, possibly requiring background domain knowledge. This type of information is often not stated explicitly in the sentence, but introduced earlier in the discourse and would hence require document-level modeling.
7.1 Entity Extraction Evaluation on the Synthesis Procedures Dataset
As described in Section 2, the data set curated by Mysore et al. (2019) contains 230 synthesis procedures annotated with entity type information. $^{11}$  We apply our models to this entity extraction task in order to estimate the degree of transferability of our findings to similar data sets. To the best of
Table 7: Experiments: slot identification. Model comparison in terms of macro F1.
CRF	BiLSTM
SciBERT	count
AnodeMaterial	25.0	19.0	280
CathodeMaterial	11.8	28.9	259
Device	59.3	67.6	381
ElectrolyteMaterial	20.0	47.2	219
FuelUsed	45.9	55.5	159
InterlayerMaterial	0.0	10.7	51
OpenCircuitVoltage	43.5	84.3	44
PowerDensity	69.0	97.6	175
Resistance	64.5	93.9	136
WorkingTemperature	72.5	90.3	414
Table 8: Experiments: slot identification. Results in terms of F1 on the test set, BiLSTM results averaged across 5 models.
Model	micro-avg. F1
DCNN (Mysore et al., 2017)	77.5
BiLSTM-CRF (Mysore et al., 2017)	77.6
BiLSTM mat2vec	73.9
BiLSTM word2vec	76.4
+ mat2vec	83.5
BERT-base	85.5
SciBERT	87.2
BiLSTM BERT-base	89.3
BiLSTM SciBERT	90.7
BiLSTM + all (with BERT-base)	89.3
BiLSTM + all (with SciBERT)	92.2
Table 9: Experiments: modeling mention types in synthesis procedure data set. Results from Mysore et al. (2017) are not directly comparable to ours as they are based on a slightly different data set; our BiLSTM mat2vec+word2vec roughly corresponds to their BiLSTM-CRF model.
our knowledge, there have not yet been any publications on the automatic modeling of this data set. We hence compare to the previous work of Mysore et al. (2017), who perform action graph induction on a similar data set. $^{12}$  Our implementation of BiLSTM-CRF mat2vec+word2vec roughly corresponds to their BiLSTM-CRF system.
Table 9 shows the performance of our models when trained and evaluated on the synthesis procedures dataset. Detailed scores by entity type can be found in the Supplementary Material. We chose to use the data split suggested by the authors for the NER task, using 200 documents for training, and 15 documents for each dev and test set. Among the non-BERT-based systems, the BiLSTM variant using both mat2vec and word2vec performs best, indicating that the two pre-trained embeddings contain complementary information with regard to this
11See https://github.com/olivettigroup/ annotated-materials-syntheses
12 According to correspondence with authors.
task. The best performance is reached by the BiLSTM model including word2vec, mat2vec, bpe and SciBERT embeddings, with 92.2 micro-average F1 providing a strong baseline for future work.
We have presented a new dataset for information extraction in the materials science domain consisting of 45 open-access scientific articles related to solid oxide fuel cells. Our detailed corpus and interannotator agreement studies highlight the complexity of the task and verify the high annotation quality. Based on the annotated structures, we suggest three information extraction tasks: the detection of experiment-describing sentences, entity mention recognition and typing, and experiment slot filling. We have presented various strong baselines for them, generally finding that BERT-based models outperform other model variants. While some categories remain challenging, overall, our models show solid performance and thus prove that this type of data modeling is feasible and can lead to systems that are applicable in production settings. Along with this paper, we make the annotation guidelines and the annotated data freely available.
Outlook. In Section 7.1, we have shown that our findings generalize well by applying model architectures developed on our corpus to another dataset. A natural next step is to combine the datasets in a multi-task setting to investigate to what extent models can profit from combining the information annotated in the respective datasets. Further research will investigate the joint modeling of entity extraction, typing and experiment frame recognition. In addition, there are also further natural language processing tasks that can be researched using our dataset. They include the detection of events and sub-events when regarding the experiment-descriptions as events, and a more linguistically motivated evaluation of the frame-semantic approach to experiment descriptions in text, e.g., moving away from the one-experiment-per-sentence and one-sentence-per-experiment assumptions and modeling the graph-based structures as annotated.
We thank Jannik Strötgen, Felix Hildebrand, Dragan Milchevski and everyone else involved in the Bosch MatKB project for their support of this research. We also thank Stefan Grünewald, Sherry Tan, and the anonymous reviewers for their insightful comments related to this paper.
Heike Adel, Laura Ana Maria Bostan, Sean Papay, Sebastian Padó, and Roman Klinger. 2018. DERE: A task and domain-independent slot filling framework for declarative relation extraction. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 42-47, Brussels, Belgium. Association for Computational Linguistics.
Emily Alsentzer, John Murphy, William Boag, WeiHung Weng, Di Jindi, Tristan Naumann, and Matthew McDermott. 2019. Publicly available clinical BERT embeddings. In Proceedings of the 2nd Clinical Natural Language Processing Workshop, pages 72-78, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
Soren Auer, Viktor Kovtun, Manuel Prinz, Anna Kasprzik, Markus Stocker, and Maria Esther Vidal. 2018. Towards a knowledge graph for science. In Proceedings of the 8th International Conference on Web Intelligence, Mining and Semantics, WIMS 18, New York, NY, USA. Association for Computing Machinery.
Isabelle Augenstein, Mrinal Das, Sebastian Riedel, Lakshmi Vikraman, and Andrew McCallum. 2017. SemEval 2017 task 10: ScienceIE - extracting keyphrases and relations from scientific publications. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 546-555, Vancouver, Canada. Association for Computational Linguistics.
Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. SciBERT: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3615-3620, Hong Kong, China. Association for Computational Linguistics.
Davide Buscaldi, Danilo Dess, Enrico Motta, Francesco Osborne, and Diego Reforgiato Recupero. 2019. Mining scholarly data for fine-grained knowledge graph construction. In Proceedings of the Workshop on Deep Learning for Knowledge Graphs, pages 21-30.
Jacob Cohen. 1968. Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological bulletin, 70(4):213.
Kevin Bretonnel Cohen, Dina Demner-Fushman, Sophia Ananiadou, and Junichi Tsujii, editors. 2017. BioNLP 2017. Association for Computational Linguistics, Vancouver, Canada..
Dina Demner-Fushman, Kevin Bretonnel Cohen, Sophia Ananiadou, and Junichi Tsujii, editors. 2018. Proceedings of the BioNLP 2018 workshop. Association for Computational Linguistics, Melbourne, Australia.
Dina Demner-Fushman, Kevin Bretonnel Cohen, Sophia Ananiadou, and Junichi Tsujii, editors. 2019. Proceedings of the 18th BioNLP Workshop and Shared Task. Association for Computational Linguistics, Florence, Italy.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Charles J. Fillmore. 1976. Frame semantics and the nature of language*. Annals of the New York Academy of Sciences, 280(1):20-32.
Silvana Hartmann, Ilia Kuznetsov, Teresa Martin, and Iryna Gurevych. 2017. Out-of-domain FrameNet semantic role labeling. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, pages 471-482, Valencia, Spain. Association for Computational Linguistics.
Benjamin Heinzerling and Michael Strube. 2018. BPEmb: Tokenization-free pre-trained subword embeddings in 275 languages. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional LSTM-CRF models for sequence tagging. In CorRR, volume abs/1508.01991.
Tianwen Jiang, Tong Zhao, Bing Qin, Ting Liu, Nitesh V. Chawla, and Meng Jiang. 2019. The role of "condition": A novel scientific knowledge graph representation and construction model. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1634-1642, Anchorage, AK, USA. ACM.
Chloe Kiddon, Ganesa Thandavam Ponnuraj, Luke Zettlemoyer, and Yejin Choi. 2015. Mise en place: Unsupervised interpretation of instructional recipes. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 982-992, Lisbon, Portugal. Association for Computational Linguistics.
Edward Kim, Kevin Huang, Alex Tomala, Sara Matthews, Emma Strubell, Adam Saunders, Andrew McCallum, and Elsa Olivetti. 2017. Machinelearned and codified synthesis parameters of oxide materials. Scientific data, 4:170127.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Jan-Christoph Klie, Michael Bugert, Beto Boullosa, Richard Eckart de Castilho, and Iryna Gurevych. 2018. The INCEpTION platform: Machine-assisted and knowledge-oriented interactive annotation. In Proceedings of the 27th International Conference on Computational Linguistics: System Demonstrations, pages 5-9, Santa Fe, New Mexico. Association for Computational Linguistics.
Olga Kononova, Haoyan Huo, Tanjin He, Ziqin Rong, Tiago Botari, Wenhao Sun, Vahe Tshitoyan, and Gerbrand Ceder. 2019. Text-mined dataset of inorganic materials synthesis recipes. Scientific data, 6(1):1-11.
Meghana Kshirsagar, Sam Thomson, Nathan Schneider, Jaime Carbonell, Noah A. Smith, and Chris Dyer. 2015. Frame-semantic role labeling with heterogeneous annotations. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 218-224, Beijing, China. Association for Computational Linguistics.
John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of the Eighteenth International Conference on Machine Learning (ICML 2001), Williams College, Williamstown, MA, USA, June 28 - July 1, 2001, pages 282-289. Morgan Kaufmann.
Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 260-270, San Diego, California. Association for Computational Linguistics.
J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. Biometrics, 33(1):159-174.
Lukas Lange, Heike Adel, and Jannik Strötgen. 2019. NLNDE: The neither-language-nor-domain-experts way of spanish medical document de-identification. In Proceedings of the Iberian Languages Evaluation Forum.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. 2018. Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3219-3232, Brussels, Belgium. Association for Computational Linguistics.
Matteo Manica, Christoph Auer, Valery Weber, Federico Zipoli, Michele Dolfi, Peter W. J. Staar, Teodoro Laino, Costas Bekas, Akihiro Fujita, Hiroki Toda, Shuichi Hirose, and Yasumitsu Orii. 2019. An information extraction and knowledge graph platform for accelerating biochemical discoveries. CoRR, abs/1907.08400.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In 1st International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings.
David Mrdjenovich, Matthew K. Horton, Joseph H. Montoya, Christian M. Legaspi, Shyam Dwaraknath, Vahe Tshitoyan, Anubhav Jain, and Kristin A. Persson. 2020. propnet: A knowledge graph for materials science. Matter, 2(2):464 - 480.
Sheshera Mysore, Zachary Jensen, Edward Kim, Kevin Huang, Haw-Shiuan Chang, Emma Strubell, Jeffrey Flanigan, Andrew McCallum, and Elsa Olivetti. 2019. The materials science procedural text corpus: Annotating materials synthesis procedures with shallow semantic structures. In Proceedings of the 13th Linguistic Annotation Workshop, pages 56-64, Florence, Italy. Association for Computational Linguistics.
Sheshera Mysore, Edward H Kim, Emma Strubell, Ao Liu, Haw-Shiuan Chang, Srikrishna Kompella, Kevin Huang, Andrew McCallum, and Elsa Olivetti. 2017. Automatically extracting action graphs from materials science synthesis procedures. In NIPS Workshop on Machine Learning for Molecules and Materials.
Vivi Nastase, Benjamin Roth, Laura Dietz, and Andrew McCallum, editors. 2019. Proceedings of the Workshop on Extracting Structured Knowledge from Scientific Publications. Association for Computational Linguistics, Minneapolis, Minnesota.
Rrubaa Panchendrarajan and Aravindh Amaresan. 2018. Bidirectional LSTM-CRF for named entity recognition. In Proceedings of the 32nd Pacific Asia Conference on Language, Information and Computation, Hong Kong. Association for Computational Linguistics.
Michael Roth and Mirella Lapata. 2015. Context-aware frame-semantic role labeling. Transactions of the Association for Computational Linguistics, 3:449-460.
Jannik Strötgen, Trung-Kien Tran, Annemarie Friedrich, Dragan Milchevski, Federico Tomazic, Anika Marusczyk, Heike Adel, Daria Stepanova, Felix Hildebrand, and Evgeny Kharlamov. 2019. Towards the bosch materials science knowledge base. In Proceedings of the ISWC 2019 Satellite Tracks (Posters & Demonstrations, Industry, and Outrageous Ideas), volume 2456 of CEUR Workshop Proceedings, pages 323-324, Auckland, New Zealand. CEUR-WS.org.
Cong Sun and Zhihao Yang. 2019. Transfer learning in biomedical named entity recognition: An evaluation of BERT in the PharmaCoNER task. In Proceedings of The 5th Workshop on BioNLP Open Shared Tasks, pages 100-104, Hong Kong, China. Association for Computational Linguistics.
Swabha Swayamdipta, Sam Thomson, Kenton Lee, Luke Zettlemoyer, Chris Dyer, and Noah A. Smith. 2018. Syntactic scaffolds for semantic structures. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3772-3782, Brussels, Belgium. Association for Computational Linguistics.
Vahe Tshitoyan, John Dagdelen, Leigh Weston, Alexander Dunn, Ziqin Rong, Olga Kononova, Kristin A. Persson, Gerbrand Ceder, and Anubhav Jain. 2019. Unsupervised word embeddings capture latent knowledge from materials science literature. Nature, 571:95 - 98.
L. Weston, V. Tshitoyan, J. Dagdelen, O. Kononova, A. Trewartha, K. A. Persson, G. Ceder, and A. Jain. 2019. Named entity recognition and normalization applied to large-scale information extraction from the materials science literature. Journal of Chemical Information and Modeling, 59(9):3692-3702. PMID: 31361962.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R'emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. HuggingFace's transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771.
A Background on Solid Oxide Fuel Cells