SlowGuess/ACL_2020_OCR huggingface dataset:
A fuel cell is an electrochemical device that generates electricity exploiting the chemical reaction of a fuel (usually hydrogen) with an oxidant (usually air). The reactions take place on two electrodes, the cathode and the anode, while the circuit is closed by an electrolyte material that only allows the transfer of charged atoms (see Figure 2). Fuel
cells that use a solid oxide as electrolyte (Solid Oxide Fuel Cells or SOFCs) are very efficient and cost-effective, but can only operate at high temperatures (500-1000C), which can cause long start-up times and fast degradation. SOFCs can be used as stationary stand-alone devices, to produce clean power for residential or industrial purposes, or integrated with other power generation systems to increase the overall efficiency.
Figure 2: Solid Oxide Fuel Cell schema.
B Data Analysis: Between-Experiment Links
As stated in Section 3, we instructed annotators to mark the closest experiment-evoking word as EXPERIMENT and link the respective slot arguments to this mention. In addition, the EXPERIMENT annotations could then be linked either by same_exp or expvariation links. Table 10 shows some statistics on the number of EXPERIMENT annotations per sentence and how often the primary annotator actually made use of the possibility to link experiments. In the training data, out of 703 sentences describing experiments, 135 contain more than one experiment-evoking word, with 114 sentences containing two, 18 sentences containing three, and 3 sentences containing four EXPERIMENT annotations (see Table 10). In the 114 sentences containing two experiment annotations, only in 2 sentences, the EXPERIMENTS were not linked to any others. Upon being shown these cases, our primary annotator judged that one of them should actually have been linked.
Next, we analyze the number of cross-sentence links. In the training data, there are 256 same_exp and 93 expvariation links, of which 138 and 57 cross sentence-boundaries respectively. Cross-sentence links between experiment-evoking words and slot fillers rarely occur in our dataset (only 13 out of 2,540 times).
# EXPERIMENT
per sentence	1	2	3	4
# sentences	568	114	18	3
# same_exp	0	82	28	7
# exp_variation	0	27	8	1
# sent. with ‘unlinked’ exp.	-	2	1	0
Table 10: Data analysis. Number of EXPERIMENT annotations per sentence, and counts of links between them (within sentence). Training set: 703 experiment-describing sentences.
C Inter-annotator Agreement Study: further statistics
Table 11 shows the full set of statistics for the experiment slot agreement.
In the following tables, we give detailed statistics for the experiments described in the main paper.
Table 12 reports full statistics for the task of identifying experiment-describing sentences, including precision and recall in the dev setting.
Table 13 reports F1 per entity type for the dev setting including standard deviations.
Table 14 reports F1 per entity type/slot for the synthesis procedures dataset (Mysore et al., 2019).
agreement study	IAA count	train count
P	R	F1
AnodeMaterial	75.0	69.2	72.0	13	280
CathodeMaterial	84.8	88.6	86.7	44	259
Conductivity	-	-	-	-	55
CurrentDensity	100.0	60.0	75.0	5	65
DegradationRate	100.0	100.0	100.0	2	19
Device	97.1	93.0	95.0	71	381
ElectrolyteMaterial	78.9	93.8	85.7	48	219
FuelUsed	90.0	81.8	85.7	11	159
InterlayerMaterial	100.0	56.0	71.8	25	51
OpenCircuitVoltage	90.0	90.0	90.0	10	44
PowerDensity	100.0	85.1	92.0	47	175
Resistance	100.0	100.0	100.0	26	136
SupportMaterial	75.0	37.5	50.0	8	106
TimeOfOperation	83.3	100.0	90.9	5	47
Voltage	100.0	33.3	50.0	6	35
WorkingTemperature	98.6	94.5	96.5	73	414
Table 11: Inter-annotator agreement study. Precision, recall and F1 scores of the two annotators vs. each other on the set of slots. IAA count refers to the number of mentions labeled with the respective type by our primary annotator in the 6 documents of the inter-annotator agreement study. train count refers to the number of instances in the training set. (Conductivity has been added to the set of slots only after conducting the inter-annotator agreement study.)
Model	dev (5-fold cv)	test
P	R	F1	P	R	F1
RBF SVM	66.4	46.1	54.2+/-3.7	64.6	54.9	59.4
Logistic Regression	72.7	41.9	53.0+/-4.2	68.2	50.9	58.3
BiLSTM mat2vec	46.3	55.6	49.9+/-3.1	49.6	69.4	57.8
BiLSTM word2vec	50.0	56.1	52.3+/-4.6	51.1	65.3	57.4
+ mat2vec	59.8	53.6	55.9+/-4.2	52.0	59.0	55.3
+bpe	62.2	56.4	58.6+/-3.0	58.9	64.7	61.7
+BERT	66.1	67.8	66.8+/-4.9	60.2	71.7	65.4
+SciBERT	68.6	68.0	68.1+/-3.7	60.2	73.4	66.1
BiLSTM BERT	65.5	64.2	64.7+/-4.6	63.7	69.9	66.7
BiLSTM SciBERT	67.1	69.1	67.9+/-4.0	58.6	74.6	65.6
BERT-base	64.0	68.2	66.0+/-4.6	58.6	71.1	64.2
BERT-large	61.8	68.9	64.3+/-4.6	63.1	75.1	68.6
SciBERT	66.0	70.2	67.9+/-4.0	60.8	74.6	67.0
humans (on agreement data)	80.4	77.6	78.9	80.4	77.6	78.9
Table 12: Experiments: Identifying experiment sentences. P, R and F1 for experiment-describing sentences. With the exception of SVM, we downsample the non-experiment-describing sentences by 0.3.
Model	EXPERIMENT	MATERIAL	VALUE	DEVICE	macroAvg.	EXPERIMENT	MATERIAL	VALUE	DEVICE	macroAvg.
CRF	66.5+/-3.5	47.0+/-9.1	73.0+/-6.4	56.2+/-10.0	60.7+/-4.5	61.4	42.3	73.6	64.1	60.3
BiLSTM mat2vec	52.9+/-3.4	55.3+/-2.0	47.9+/-6.3	53.2+/-1.9	52.3+/-3.4	47.1	52.4	60.9	46.1	51.6
+ BERT	80.3+/-3.2	87.7+/-3.3	76.8+/-5.3	81.9+/-5.5	81.7+/-4.3	74.3	87.9	71.0	80.7	78.5
BiLSTM word2vec	62.3+/-3.0	61.6+/-2.1	52.1+/-5.2	59.5+/-1.0	58.9+/-2.8	55.8	58.6	59.1	51.7	56.3
+mat2vec	65.8+/-4.2	78.4+/-1.6	61.9+/-8.2	69.6+/-4.0	68.9+/-4.5	57.9	75.2	64.3	61.5	64.7
+bpe	69.2+/-5.8	82.3+/-1.9	60.1+/-11.2	73.4+/-4.7	71.2+/-5.9	63.3	81.6	68.0	68.1	70.2
+BERT	80.0+/-3.4	87.9+/-2.8	74.4+/-5.6	80.7+/-3.9	80.8+/-3.9	76.0	88.1	72.9	81.5	79.7
+SciBERT	81.4+/-1.6	89.4+/-2.4	73.8+/-8.7	82.0+/-4.3	81.7+/-4.2	76.9	89.8	74.1	85.2	81.5
BiLSTM BERT	79.6+/-2.4	87.6+/-2.4	72.0+/-7.5	80.5+/-5.1	79.9+/-4.3	75.4	87.6	72.6	80.8	79.1
BiLSTM SciBERT	80.5+/-1.2	89.4+/-2.8	73.0+/-9.4	82.3+/-3.5	81.3+/-4.2	77.1	89.9	72.1	85.7	81.2
BERT-base	85.4+/-2.8	73.7+/-7.2	90.0+/-2.1	68.3+/-3.7	79.3+/-3.9	81.8	70.6	88.2	73.1	78.4
SciBERT	84.5+/-3.0	77.0+/-7.4	91.6+/-2.8	72.7+/-2.1	81.5+/-3.8	81.2	75.3	91.9	73.2	80.4
humans	94.3	95.9	95.5	97.5	95.8	94.3	95.9	95.5	97.5	95.8
Table 13: Experiments: entity mention extraction and labeling. Results on 5-fold cross validation for dev and test set (experiment-describing sentences only) in terms of F1.
Entity Types	Mysore et al. (2017)	BiLSTM w2v+m2v	BiLSTM + all (SciBERT)
Amount-Unit	83.5	93.5	95.8
Brand	-	67.9	83.3
Condition-Misc	74.6	85.1	88.9
Condition-Unit	94.5	97.2	95.0
Material	80.2	84.0	92.3
Material-Descriptor*	62.0	65.5	88.5
Nonrecipe-Material	-	45.8	80.0
Number	91.9	93.4	98.4
Operation	82.8	93.5	98.1
Synthesis-Apparatus	-	63.9	81.3
Table 14: Experiments: Modeling mention types in synthesis procedure data, most frequent entity types. Results in terms of F1. Results from Mysore et al. (2017) are not directly comparable. *Type called Descriptor in their paper.