SlowGuess/ACL_2020_OCR huggingface dataset:
where  $\gamma$  is a scaling factor,  $c_{i}^{j}$  is the annotator confidence for annotation  $j$  on token  $i$ ,  $\nu$  is the set of predicted attributes, and  $\nu^{*}$  is the set of true attributes. Note that inclusion of the confidence mask  $c_{i}^{j}$  means the model only incurs loss on attributes annotated for a given node, since  $c_{i}^{j} = 0$  when an annotation is missing (i.e. no MSE loss is incurred for attributes which do not apply to a node or edge); in the "binary" experimental setting, we replace  $c_{i}^{j}$  with  $\tau(c_{i}^{j})$ , removing the weighting but still masking out loss on un-annotated nodes. Also note that in the case of edges, the form of the loss is identical, but  $\nu$  is replaced by  $\lambda$ , and  $\alpha$  by  $\beta$ .
This loss encourages the predicted attribute  $\nu_{i}^{j}$  value to be close in value to the true value  $\nu_{i}^{j*}$  via the mean-squared error criterion while concomitantly encouraging the predicted and reference values to share a sign via the thresholded cross-entropy criterion. Both node and edge attribute models are trained to predict attribute values independently, and that parameters are shared across attributes. This is central to our analysis in  $\S 7$ .
Following Zhang et al. (2019b) we train the structural parsing modules with coverage loss (See et al., 2017). All models were trained to convergence using the Adam optimizer (Kingma and Ba, 2014) with a learning rate of 0.001.
Pipeline Model Recall from Section 3 that the semantic graph structure in UDS graphs is deterministically generated from PredPatt, which takes as input a UD parse and outputs a semantic graph structure. This leads to a strong pipeline model for the graph structure alone: running a high-performing UD parser—the Stanford UD parser (Chen and Manning, 2014)—and passing its output through PredPatt to create a structure. For this baseline,
Method	P	R	F1
Pipeline	84.83	75.22	79.74
Parser	83.52	77.92	80.62
Parser (binary)	84.97	78.52	81.62
Table 2: Test set  $S$  score precision, recall, and F1.
the only source of error is the UD parsing model, which for English performs very highly.
S Metric For evaluating the quality of output graph structures, Smatch (Cai and Knight, 2013), a hill-climbing approach to approximating the optimal matching between variables in two graphs, is commonly used. While Smatch can match categorical variables such as those found in meaning representations like AMR, it lacks a matching function for continuous variables such as decompositional attributes. To remedy this, Zhang et al. (2018) introduce the  $S$  metric, an extension to Smatch that allows for attribute matching.
Using hill-climbing, we are able to match instance and attribute nodes and edges; instance nodes are matched via string match, while attribute similarity is given by  $1 - \left(\frac{\nu_i - \nu_j}{\omega}\right)^2$  where  $\omega = 6$  is the maximum possible difference between attributes, which are bounded on  $[-3,3]$ .
Table 5 shows the Pearson's correlation coefficient  $(\rho)$  and the F1 score computed on binarized responses for each node and edge attribute under the "oracle" decoding setting, where a gold graph structure is provided to the model. An asterisk denotes that  $p < 0.05$ , where  $p$  is determined by a Student's t-test. F1 scores are obtained by binarizing continuous attribute predictions into positive and negative, following from the original UDS motivation found in Dowty (1991), where binary proto-role features were introduced. The binarization threshold was tuned per attribute on the validation set.
The baseline column in Table 5 shows the binarized F1 score for the baseline attribute model, given by predicting the median attribute value for each attribute type at each position. Pearson's  $\rho$  is undefined for this approach, as the variance of the predicted distribution is 0. The thresholds were similarly tuned on validation data for this baseline.
Table 2 shows S metric (c.f. §5) precision, recall, and F1 score as computed on full arborescences
This structure is missing the core decompositional attributes but has both predicate and argument nodes. Additionally, the pipeline model fails to capture nominal heads of copular predicates (e.g. Jo is a doctor), which are not returned by PredPatt but are added to the dataset as a preprocessing step in the genericity annotation task.
This function was found to produce more matches on UDS1.0 than the  $e^{-\mathrm{MAE}}$  function used by Zhang et al. (2018).
8432
with both semantics and syntax nodes. Our parser slightly outperforms the pipeline, with higher performance in the "binary" setting, where we exclude annotator confidence from the loss.
Table 3 shows precision, recall, and F1 score on semantics nodes alone. The first parser setting (syntax) reflects a parsing model trained on full graphs, and evaluated only on the semantic subgraphs of the produced graphs. The second parser (semantics) is directly trained on semantic subgraphs, with no syntactic nodes in the training graphs. The full parser performs comparably to the pipeline, while the parser trained specifically on semantics-only graphs outperforms the pipeline. However, the mean attribute  $\rho$  of the syntactic parser (0.3433) exceeded that of the semantics-only parser (0.3151).
Method	P	R	F1
Pipeline	84.72	88.51	86.57
Parser (syntax)	89.02	83.67	86.26
Parser (syntax, binary)	89.74	86.00	87.83
Parser (semantics)	91.28	87.23	89.21
Parser (sem., binary)	91.10	84.59	87.73
Table 3: Test set  $S$  score precision, recall, and F1 on semantics nodes only, where (syntax) denotes a parser trained to predict full graphs (semantics nodes with non-head edges to syntax nodes) while (semantics) denotes model trained on semantics-only subgraphs.
Table 4 gives the  $S$  metric results on full graphs predicted by the model, including attribute matching. The pipeline model is unable to perform this task because it predicts structure alone, without attributes. We see that training the parser with shared MLP and BILINEAR modules (i.e.  $\mathrm{MLP}^{(\mathrm{mask})} = \mathrm{MLP}^{(\mathrm{attr})}$  and  $\mathrm{BILINEAR}^{(\mathrm{mask})} = \mathrm{BILINEAR}^{(\mathrm{attr})}$ ) for both the attribute mask and attribute value heavily degrades the performance, while removing annotator confidence increases it slightly.
Table 2 suggests that the structural quality of the parses obtained by the parsing model presented here is slightly superior to that of pipeline model's parses, with Table 3 indicating that the semantic component of the graph can be parsed significantly more accurately by our model. Taken together with Table 5, we can conclude that the model is able to learn to jointly predict the graph structure and attributes. This is further reinforced by Table 4. Note that the numbers reported in Tables 2 and 4 are not directly comparable, as the scores in Table 4
Method	P	R	F1
Shared	79.52	32.48	46.12
Separate	83.46	82.27	82.86
Separate (binary)	84.19	84.19	84.19
Table 4: Test set precision, recall, and F1 computed via  $S$  score with attributes (syntactic nodes included)
Property	Pearson's ρ (model)	F1 (baseline)	F1 (model)
node-level	genericity	factuality-factual	0.6479*	75.15
arg-abstract	0.3392*	40.04
arg-kind	0.2145*	67.61
arg-particular	0.3347*	83.10
pred-dynamic	0.2469*	72.49
pred-hypothetical	0.3442*	44.16
pred-particular	0.1887*	77.47
time	dur-centuries	0.1336*	10.14
dur-days	0.1802*	68.72
dur-decades	0.2383*	29.89
dur-forever	0.2524*	37.93
dur-hours	0.2227*	73.66
dur-instant	0.1761*	55.98
dur-minutes	0.3409*	86.28
dur-months	0.3204*	63.25
dur-seconds	0.2751*	65.33
dur-weeks	0.2475*	54.02
dur-years	0.4239*	65.03
wordsense	supersense-noun.Top	0.4660*	7.34
supersense-noun.act	0.6007*	27.37
supersense-noun animal	0.3773*	5.60
supersense-noun.artifact	0.5617*	23.12
supersense-noun_attribute	0.4505*	10.81
supersense-noun.body	0.4543*	1.53
supersense-noun.cognition	0.5692*	21.17
supersense-noun.communication	0.6182*	30.60
supersense-noun.event	0.4233*	5.80
supersense-noun feelsing	0.2404*	2.74
supersense-noun.food	0.6773*	7.15
supersense-noun.group	0.5650*	15.57
supersense-noun.location	0.5118*	7.81
supersense-noun.motive	0.3447*	0.62
supersense-noun.object	0.2276*	2.04
supersense-noun.person	0.6091*	15.74
supersense-noun.phenomenon	0.2955*	2.04
supersense-noun.plant	0.0358	0.21
supersense-noun.possession	0.5247*	6.67
supersense-noun-process	0.1292*	1.13
supersense-nounquantity	0.4403*	4.92
supersense-noun.relation	0.2089*	2.34
supersense-noun.shape	0.0659*	0.31
supersense-noun.state	0.4877*	11.36
supersense-noun.substance	0.2411*	1.43
supersense-noun.time	0.5175*	10.99
protoroles	awareness	0.6715*	68.20
change-of-location	0.1061*	38.98
change-of-possession	0.0452	14.93
change-of-state	0.0448	42.59
change-of-state-continuous	0.0793	31.47
existed-after	0.3910*	93.33
existed-before	0.4802*	91.60
existed-during	0.3247*	98.31
instigation	0.3820*	74.48
partitive	0.0213	31.91
sentient	0.6494*	64.67
volition	0.5501*	63.79
was-for-benefit	0.2389*	59.87
was-used	0.1608*	86.64
macro-average	0.3433	37.20	50.66
Table 5: Pearson's  $\rho$  ,baseline F1,and model F1 for each UDS attribute given gold test-set graph structures.
incorporate the matching scores between attributes.
Table 3 shows that a parser trained on semantic subgraphs better recovers the subgraphs than a parser trained on full graphs whose outputs are
8433
postprocessed to remove syntactic nodes. However, the fact that the parser trained on full graphs achieves a higher Pearson's  $\rho$  score indicates that the inclusion of syntactic nodes may provide additional information for predicting UDS attributes.
In examining instances with an  $S$  score below 50, we observe two trends: the input sentences are often ungrammatical, and for  $63.82\%$  (on the validation set) the model predicts no output nodes.
While the pipeline system does well on modeling semantic graph structure, it is by its definition unable to perform attribute parsing. In contrast, the results presented in Tables 4 and 5 show that the parser can jointly learn to produce semantic graphs and annotate them with attributes.
Finally, we find that while weighting the loss with the confidence scores has a small benefit in the semantics-only setting, it hurts overall attribute and structure prediction performance. This may be due to the relatively small size of the UDS dataset, which makes a strategy that is effectively weakening the loss signal at training time less effective.[6]
Figs. 3a-3c show the correlational strength coefficient between the true and predicted attributes under a forced decode of the graph structure. It is defined over property types indices  $j$ ,  $k$  with predicted attribute values  $\nu_{i}^{j}$  and true values  $\nu_{i}^{j*}$  as:
$$
\psi (j, k) = \tanh  \left(1 - \frac {\left| \operatorname {c o r r} \left(\nu^ {j} - \nu^ {j *} , \nu^ {k} - \nu^ {k *}\right) \right|}{\left| \operatorname {c o r r} \left(\nu^ {j *} , \nu^ {k *}\right) \right|}\right)
$$
where  $\mathrm{corr}(\nu^{j*},\nu^{k*})$  is Pearson's correlation coefficient. Further details are given in Appendix A.
$\psi (i,j)$  reflects how well the model captures the strength of the correlations (either positive or negative) between two attribute types in the dataset: a positive value indicates that the model captures the correlation to some extent, with values closer to 1 implying better performance; a value of 0 indicates that the model does not capture the correlation at all, or that no significant interaction was present; a negative value indicates that the model makes systematic mistakes while predicting the two variables, e.g. when the model under-predicts the value of property  $i$ , it also under-predicts property  $j$ 's value. A Bonferroni-corrected non-parametric bootstrap test (1000 replicants) was used for significance testing, with failing pairs being said to not be reliably different from 0 correlation.
Fig. 3a shows the model typically systematically under- or over-predicts the values for pairs
(a)  $\psi$  between argument-node attribute pairs. Subset of word-senses used for readability.
(b)  $\psi$  between predicate-node attribute pairs. Subset of time attributes used for readability.
(c)  $\psi$  between all edge attribute pairs.
Figure 3:  $\psi$  heatmaps for UDS1.0 attribute pairs of argument-node attributes, with most  $\psi$  values close to -1. However, we do see positive correlations between some of the genericity annotations,
6All confidence are on [0, 1]
8434
Sentences	Property	(A)	Ours (A)	(B)	Ours (B)	(C)	Ours (C)
(A) She was untrained and, in one botched job, killed a client.	awareness	3	3.04	1	3.69	5	3.68
volition	2	2.92	1	3.45	5	3.44
(B) The antibody then kills the cell.	instigation	5	3.08	5	3.39	5	3.37
(C) An assassin in Colombia killed a federal judge on a Medellin street.	sentience	5	2.99	1	3.71	5	3.70
existed-after	5	3.57	2	3.79	5	3.78
Table 6: Comparison of gold properties from Reisinger et al. (2015) (on an ordinal scale from 1 to 5, with 3 as "neutral") and predicted properties (mapped to [1, 5]) for sentences involving the predicate "kills".
as well as between genericity-arg-abstract, which rates how conceptually abstract an argument is, and the cognition wordsense, which applies to abstract terms such as "doubts" and "thoughts".
In Fig. 3b, we again observe several negative  $\psi$  values; however, some positive correlations can be seen between certain time properties, such as duration-days, duration-weeks, and duration-months, as well as more strongly positive  $\psi$ 's between certain genericity annotations. The positive  $\psi$  between factuality and genericity-hypothetical indicates the model has captured the commonalities between predicates with these annotations.
In contrast to the node attributes, Fig. 3c shows stronger results for edge attribute prediction, with all significant  $\psi$  's being positive, and related attributes falling into clusters (e.g. volition, awareness, sentience, or the existed attributes)
Qualitative examples Table 6 lists three sentences from Reisinger et al. (2015) along with a relevant subset of their original SPR properties and values; the scale in Reisinger et al. was ordinal from 1-5, with 1 corresponding to "very unlikely," 5 to "very likely," and 3 to "neutral." Our model's predictions for the same sentences and properties are given as well, mapped onto [1, 5]. We first note that the structural component of the model is sufficiently strong that the correct predicate-argument edges were extracted during parsing, allowing for a direct comparison between the annotations by Reisinger et al. and the parser's predictions. We see that while for sentence (C), the model captures at least the correct direction of the protonole annotations, it overgeneralizes these results to (B), where a more nuanced analysis is required. For (A), we see that on most attributes the model captures the desired binary direction of the inferences, but that it fails on sentience. Overall, the model's predictions are weaker than the desired output, even when the prediction is on the correct side of the midpoint, 3. This might help explain the disparity between Pearson and F1 scores in Table 5, and represents
a direction for future work. Note that to obtain attributes for (A) and (B), the threshold for the masks  $\beta$  was dropped; ideally, this would not be required.
The scalar valued, multi-attribute nature of UDS provides for a distinct structured prediction problem as compared to other existing representations. We have demonstrated how a transductive parsing paradigm that has achieved state-of-the-art results on other representations can be adapted to UDS1.0 structures and attributes, and have provided procedures for analysis, with the fine-grained nature of UDS allowing for investigating novel correlations and aspects of meaning. While UDS structures and various attribute types have been modeled separately (Vashishtha et al., 2019; Govindarajan et al., 2019; White et al., 2016; Rudinger et al., 2018a,b; Zhang et al., 2018), this work represents the first time all of these attributes and structures have been modeled jointly, and establishes a baseline for future efforts on UDS1.0.
We envision future efforts exploring the interactions between improving the underlying graph-structure prediction and ever-better correlations to human judgements on individual properties.
This work was supported by NSF Awards #1749025 and #1763705, DARPA LORELEI and AIDA, and IARPA BETTER. We thank the anonymous reviewers for their constructive feedback. The views and conclusions expressed herein are those of the authors and should not be interpreted as representing official policies or endorsements of DARPA, IARPA, or the U.S. Government.
Omri Abend and Ari Rappoport. 2013. Universal Conceptual Cognitive Annotation (UCCA). In Proceedings of the 51st Annual Meeting of the Association
8435
for Computational Linguistics (Volume 1: Long Papers), pages 228-238, Sofia, Bulgaria. Association for Computational Linguistics.
Jacob Andreas, Andreas Vlachos, and Stephen Clark. 2013. Semantic parsing as machine translation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 47-52.
Miguel Ballesteros and Yaser Al-Onaizan. 2017. AMR parsing using stack-LSTMs. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1269-1275, Copenhagen, Denmark. Association for Computational Linguistics.
Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffith, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract meaning representation for sembanking. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pages 178-186.
Ann Bies, Justin Mott, Colin Warner, and Seth Kulick. 2012. English web treebank. Linguistic Data Consortium, Philadelphia, PA.
Deng Cai and Wai Lam. 2020. Amr parsing via graph-sequence iterative inference. arXiv preprint arXiv:2004.05572.
Shu Cai and Kevin Knight. 2013. Smatch: an evaluation metric for semantic feature structures. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 748-752, Sofia, Bulgaria. Association for Computational Linguistics.
Danqi Chen and Christopher Manning. 2014. A fast and accurate dependency parser using neural networks. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 740-750.
Marco Damonte, Shay B. Cohen, and Giorgio Satta. 2017. An incremental parser for abstract meaning representation. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 536-546. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
David Dowty. 1991. Thematic proto-roles and argument selection. Language, 67(3):547-619.
Timothy Dozat and Christopher D Manning. 2016. Deep bioaffine attention for neural dependency parsing. arXiv preprint arXiv:1611.01734.
James Goodman, Andreas Vlachos, and Jason Naradowsky. 2016. Ucl+sheffield at semeval-2016 task 8: Imitation learning for amr parsing with an alphabet bound. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 1167-1172. Association for Computational Linguistics.
Venkata Govindarajan, Benjamin Van Durme, and Aaron Steven White. 2019. Decomposing Generalization: Models of Generic, Habitual, and Episodic Statements. Transactions of the Association for Computational Linguistics, 7:501-517.
Daniel Hershcovich, Omri Abend, and Ari Rappoport. 2017. A transition-based directed acyclic graph parser for UCCA. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1127-1138, Vancouver, Canada. Association for Computational Linguistics.
Daniel Hershcovich, Omri Abend, and Ari Rappoport. 2018a. Multitask parsing across semantic representations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 373-385, Melbourne, Australia. Association for Computational Linguistics.
Daniel Hershcovich, Omri Abend, and Ari Rappoport. 2018b. Universal dependency parsing with a general transition-based dag parser. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 103-112.
Daniel Hershcovich, Zohar Aizenbud, Leshem Choshen, Elior Sulem, Ari Rappoport, and Omri Abend. 2019. SemEval-2019 task 1: Cross-lingual semantic parsing with UCCA. In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 1-10, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
Wei Jiang, Zhenghua Li, Yu Zhang, and Min Zhang. 2019. Hlt@suda at semeval 2019 task 1: Ucca graph parsing as constituent tree parsing. arXiv preprint arXiv:1903.04153.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Jonathan May. 2016. SemEval-2016 task 8: Meaning representation parsing. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 1063-1073, San Diego, California. Association for Computational Linguistics.
8436