SlowGuess/ACL_2020_OCR huggingface dataset:
Elias Stengel-Eskin Johns Hopkins University
Sheng Zhang Johns Hopkins University
Aaron Steven White  
University of Rochester
Benjamin Van Durme Johns Hopkins University
We introduce a transductive model for parsing into Universal Decompositional Semantics (UDS) representations, which jointly learns to map natural language utterances into UDS graph structures and annotate the graph with decompositional semantic attribute scores. We also introduce a strong pipeline model for parsing into the UDS graph structure, and show that our transductive parser performs comparably while additionally performing attribute prediction. By analyzing the attribute prediction errors, we find the model captures natural relationships between attribute groups.
A structured account of compositional meaning has been longstanding goal for both natural language understanding and computational semantics. To this end, a number of efforts have focused on encoding semantic relationships and attributes in a semantic graph—e.g. Abstract Meaning Representation (AMR; Banarescu et al., 2013), Universal Conceptual Cognitive Annotation (UCCA; Abend and Rappoport, 2013), and Semantic Dependency Parsing (SDP; Oepen et al., 2014, 2015, 2016).
In these formalisms, semantic information is typically encoded discretely, using nominal category labels for nodes and edges. This categorical encoding can make such formalisms brittle when presented with non-prototypical instances, and leads to challenges in coping with changing label ontologies and new datasets (White et al., 2019). Furthermore, they are difficult to annotate, often requiring trained linguists and large annotation manuals.
The Decompositional Semantics framework presents an alternative to categorical formalisms that encodes semantic information in a feature-based scheme—using continuous scales rather than categorical labels. Starting with a feature-based semantic role representation rooted in Dowty 1991's
(1991) proto-role theory (Reisinger et al., 2015; White et al., 2016), this framework has expanded to cover a wide variety of phenomena: event factuality (Rudinger et al., 2018b), genericity (Govindarajan et al., 2019), entity types (White et al., 2016), and temporal relations (Vashishtha et al., 2019).
While this rich array of annotation types has been separately modeled, no system yet exists for its joint prediction, which has only recently been made feasible by the introduction of Universal Decompositional Semantics v1.0 (UDS1.0). Presented by White et al. (2019), UDS1.0 normalizes all of these annotations, and incorporates them as node- and edge-level attributes in a single semantic graph whose structure is deterministically extracted from Universal Dependencies (UD; Nivre et al., 2015) syntactic parses via the PredPatt tool (White et al., 2016; Zhang et al., 2017). An example graph can be seen in Fig. 1.
We present the first joint UDS parser, which learns to extract both graph structures and attributes from natural language input. This parser is a sequence-to-graph transductive model which takes as input a sentence and outputs a UDS graph complete with node- and edge-level annotations.
In contrast to the traditional semantic parsing paradigm, which shares its roots with syntactic parsing and rests on the assumption that the nodes in the graph correspond to tokens in the input—i.e. the graph is lexicalized—the parsing-a-transduction paradigm treats parsing as a sequence-to-graph problem. Rather than generating one sequence conditional on another sequence (sequence-to-sequence), we generate the nodes in a graph conditional on an input sequence, dynamically adding their edges during generation. As in sequence-to-sequence modeling, the supports of the input and output distributions—i.e. the input and output
Available at http://decomp.io.
8427
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8427-8439 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Figure 1: The UDS graph structure. Semantic subgraph is outlined in black while the syntactic subgraph is annotated in pink. Node and edge attribute annotations are shown via annotations on argument and attribute edges.
vocabularies--are not constrained to be identical.
This has two benefits: first, post-hoc methods of obtaining alignments between input sequences and graphs—common especially in AMR parsing—are no longer required; and second, we are able to produce semantic graphs from arbitrary input vocabularies—allowing for future extensions to cross-lingual parsing (Zhang et al., 2018). The parsing-as-transduction paradigm thus lends itself perfectly to UDS parsing, since the UDS protocol allows non-lexicalized (as well as cross-lingual) graphs, and these graphs may have nodes with multiple parents—i.e. re-entrant nodes—which pose problems for traditional tree-based methods but are handled natively by the transductive paradigm.
We compare our end-to-end transductive parser against a strong pipeline system, finding that the parser slightly outperforms the pipeline while additionally learning to produce decompositional attribute scores. Our results are reflected in the UDS1.0 leaderboard at http://decomp.io/leaderboards/.
Datasets Reisinger et al. (2015) introduce the Decompositional Semantics framework in the context of a corpus-based verification of Dowty's seminal proto-role theory of semantic roles. This work was substantially expanded by White et al. (2016), who annotate for semantic proto-roles (SPR), word-sense, and temporal properties on top of semantic graphs extracted from English Web Treebank (EWT; Bies et al., 2012) UD parses using PredPatt (White et al., 2016; Zhang et al., 2017).
White et al.'s EWT annotations are modeled by Teichert et al. (2017), who present a CRF-based multi-label classifier for proto-role labelling, and
Rudinger et al. (2018a), who make use of an event-driven neural model. More recently, the annotation coverage for the same EWT data was expanded by Vashishtha et al. (2019) who annotate and model fine-grained temporal distinctions, and Govindarajan et al. (2019), who add annotations and models for genericity—i.e. the degree of generality of events and entities in linguistic expressions.
All of these efforts coalesce in White et al. (2019), which presents the first unified Decompositional Semantics-aligned dataset—Universal Decompositional Semantics v1.0 (UDS1.0)—containing all properties annotated on top of EWT parses with standardized train, validation, and testing splits and a native reader and query interface.
Parsing In most work on decompositional semantics, models are tasked with learning to predict attribute values, but not the structure of the graph. Zhang et al. (2018) develop the first model for performing both graph parsing and UDS attribute prediction in a cross-lingual setting, where Chinese input sentences were transduced into UDS graphs derived from UD parses of the input's English translation. This represents the first application of the parsing-as-transduction paradigm to a subset of UDS data as well as the introduction of a novel graph evaluation metric,  $S$  which we describe in further detail in Section 5. In contrast to the end-to-end approach presented here, Zhang et al. take a pipeline approach to parsing.
Andreas et al. (2013) recast semantic parsing in a tree formalism as a sequence-to-sequence problem. Parsing-as-transduction, which extends this approach to directed acyclic graphs, has proven to be applicable in a variety of settings: Zhang et al. (2019a) use it to achieve state-of-the-art re
8428
sults in AMR parsing. These results are improved upon and shown to generalize to two other semantic formalisms (UCCA and SDP) by Zhang et al. (2019b), which set new state-of-the-art benchmarks for AMR and UCCA. The former result was subsequently surpassed by Cai and Lam (2020), which applies a similar transductive approach, while the latter was surpassed by Jiang et al. (2019).
Having both been subjects of SemEval tasks (May, 2016; May and Priyadarshi, 2017; Oepen et al., 2019; Hershcovich et al., 2019), there are a number of contrasting methods for both AMR and UCCA parsing. These include transition-based parsing system for AMR (Wang et al., 2018; Goodman et al., 2016; Damonte et al., 2017; Ballesteros and Al-Onaizan, 2017) and for UCCA (Hershcovich et al., 2017). In a similar vein to Zhang et al. (2019b), Hershcovich et al. (2018a) convert multiple formalisms into a unified formalism and use multitask learning for improved UCCA parsing; however, the latter does so at a loss to performance on the other formalisms, while Zhang et al. achieve state-of-the-art results in AMR and UCCA simultaneously. UCCA has also been shown to transfer to syntactic parsing: by converting UD parse trees into a format resembling UCCA, Hershcovich et al. (2018b) are able to apply a UCCA parser to both standard UD parses as well as enhanced UD parses, which contain re-entrant nodes.
The UDS1.0 dataset is built on top of the UD-EWT data with three layers of annotations: UD parses, PredPatt graph structure, and decompositional semantic annotations on the edge and node level. In addition to specifying the syntactic head and head relation of every token in the input, UD parses include lexical features, such as word form, word lemma, and part-of-speech (POS) tag. This forms the syntactic graph, which is lexicalized (each token is tied to a node in the graph). From these pieces of information, PredPatt outputs a set of predicates and their arguments.
Each predicate and argument is tied via an instance edge to a particular node in the syntactic graph. Because both predicates and arguments can consist of multi-word spans, there can be multiple instance edges leaving a semantic node. The semantic graph contains edges between predicates and arguments; in the case of clausal embedding, there can also be argument-argument edges. UDS1.0
Figure 2: Arborescence for graphs with object control.
includes "performative" speaker/author and addressee nodes, which model discourse properties of the sentence. These nodes are structural placeholders for future discourse-level annotations; as these properties have not yet been annotated, we have opted to remove them from the graphs.2
The crowdsourced decompositional annotations tied to the semantic subgraph can be divided into node-level annotations and edge-level annotations. On the node level, annotations were collected for factuality, genericity, time, and entity type. Edge-level annotations are in the space of semantic protoroles, which are designed to provide a nuanced higher-dimensional substrate for notions of agency and patienthood. These are summarized in Table 1, where purple indicates a high attribute score, while orange indicates a low score. For further details on attribute types and data annotation, see White et al. (2019) and the references therein.
Arborescence Recall that the lowest level of the UDS graph (Fig. 1) is a syntactic dependency parse. Modeling this level is out of scope for this work, as we are interesting in modeling the semantic structure and attributes. In order to train a parsing-a-transduction model, an arborescence—a hierarchical tree structure which has only edge and node annotations—is required. From the full UDS graph, we construct the arborescence by:
(a) Assigning each semantic node a lexical label; this label is taken from the syntactic head that the semantic node dominates. The only exception to this is in the case of embedded clauses, where an argument node dominates an embedded predicate. Here, we follow PredPatt, assigning the label "SOMETHING" to the embedded argument (c.f. Fig. 2).
<sup>2</sup>Since these placeholder nodes are currently added deterministically, recovering them is also a deterministic operation.
8429
Annotation	Description	Examples
Factuality	Factuality inferences represent how likely (or unlikely) a listener thinks a scenario is to have occurred.	Jo left (3), Jo didn’t leave (-3), Jo thought that Cole had left (-1)
Genericity	Genericity refers to inferences about the generality of events or event participants.	Ex. property: genericity-pred-particular: Amy ate oats for breakfast today (3), Amy ate oats for breakfast every day (-3)
Time	Temporal inferences pertain to the duration of events.	Ex. property: time-dur-minutes: Tom left (-3), Tom was singing (3)
Word Sense	UDS decomposes word sense, allowing multiple senses to apply to a given node.	Ex. property: supersense.person: Sandy led Rufus by a leash (-3), Sandy led Rufus by a leash (3)
Semantic
Proto-Roles	SPR properties are edge-level annotations that capture fine-grained semantic relations between predicates and arguments.	Ex. property: volition:, Derek broke his arm (-3), Derek broke the wishbone (3)
Table 1: Type descriptions and illustrative sentences for UDS properties predicted in this work. Example ratings in parentheses, bolding indicates the salient predicate/argument/edge. See White et al. (2019) for further details.
(b) Retaining all edges between semantic nodes as "argument" edges, duplicating nodes in cases of re-entrancy (e.g. "Bush $_{(1)}$ ") in Fig. 2).
(c) Converting the deep syntactic structure into a shallow representation, where we introduce "non-head" edges from the syntactic head (attached to a semantic node) to each node it dominates, and remove all other syntax-semantics edges. This effectively linearizes the yield of each semantic node (see Fig. 2).
Our model is based on the transductive broad-coverage parsing model presented in Zhang et al. (2019b), which can be consulted for further details on the encoder, decoder, and pointer-generator modules. The original parser is composed of six major modules: the encoder, the decoder embedding module, the target node module, the target label module, the head module, and the relation module. In this work we introduce two new modules: the node attribute module and the edge attribute module, as well a loss function for attributes.
Encoder The encoder module takes a concatenation of multiple input features: GloVe token embeddings (Pennington et al., 2014), POS tag embeddings, character CNN embeddings, and BERT (Devlin et al., 2019) contextual embeddings (mean-pooled over subwords). These representations are passed through a stacked bidirectional LSTM encoder, which has the following definition:
$$
\mathbf {s} _ {t} ^ {l} = \left[ \begin{array}{c} \overrightarrow {\mathbf {s}} _ {t} ^ {l} \\ \overleftarrow {\mathbf {s}} _ {t} ^ {l} \end{array} \right] = \left[ \begin{array}{c} \overrightarrow {\mathrm {L S T M}} (\mathbf {s} _ {t} ^ {l - 1}, \mathbf {s} _ {t - 1} ^ {t}) \\ \overleftarrow {\mathrm {L S T M}} (\mathbf {s} _ {t} ^ {l - 1}, \mathbf {s} _ {t + 1} ^ {t}) \end{array} \right]
$$
where arrows denote the LSTM direction,  $t$  denotes the timestep, and  $l$  denotes the layer of the stack.
Decoder embedding module In order to generate new semantic nodes and relationships, a method of embedding categorical semantic information is required. More formally, a semantic relation is given by a tuple  $\langle u_i,d_i^u,r_i,v_i,d_i^v\rangle$ , where  $u_{i}$  denotes the "head" token of index  $i$  and  $v_{i}$  denotes the token at index  $i$ . Note that these tokens are the labels of nodes in the arborescence (see Fig 2.)  $d_i^u$  and  $d_i^v$  are the indices of  $u_{i}$  and  $v_{i}$ , while  $r_i$  is the relationship type between  $v_{i}$  and  $u_{i}$ . The decoder embedding module embeds these categorical variables into real space, producing a tuple of vectors  $\langle \mathbf{u}_i,\mathbf{d}_i^u,\mathbf{r}_i,\mathbf{v}_i,\mathbf{d}_i^v\rangle$ . For node labels  $\mathbf{u}_i$  and  $\mathbf{v}_i$ , we take the concatenation of GloVe and CharCNN features.  $\mathbf{r}_i$ ,  $\mathbf{d}_i^v$  and  $\mathbf{d}_i^u$  are randomly initialized.
Target Node Module From the continuous embedding of a semantic relation  $\langle \mathbf{u}_i, \mathbf{d}_i^u, \mathbf{r}_i, \mathbf{v}_i, \mathbf{d}_i^v \rangle$  we want to obtain a latent node representation  $\mathbf{z}_i$ . We initialize the hidden states of the  $0^{\text{th}}$  layer and the hidden states of the  $0^{\text{th}}$  state in each layer to
$$
\mathbf {h} _ {\mathbf {i}} ^ {0} = \left[ \mathbf {v} _ {i}; \mathbf {d} _ {i} ^ {v} \right]
$$
$$
\mathbf {h} _ {\mathbf {0}} ^ {l} = \left[ \overleftarrow {\mathbf {s}} _ {1} ^ {l}; \overrightarrow {\mathbf {s}} _ {n} ^ {l} \right]
$$
respectively. Further, let  $\mathbf{c}_i$  be a context vector over encoder states  $\mathbf{s}_{1:n}^l$ , defined as
$$
\mathbf {a} _ {i} ^ {\left(\mathrm {e n c}\right)} = \operatorname {s o f t m a x} \left(\mathbf {M L P} ^ {\left(\mathrm {e n c}\right)} \left(\left[ \mathbf {h} _ {i} ^ {l}; \mathbf {s} _ {1: n} ^ {l} \right]\right)\right)
$$
$$
\mathbf {c} _ {i} = \mathbf {a} _ {i} ^ {T} \mathbf {s} _ {1: n} ^ {l}
$$
8430
Let  $\mathbf{h}_i^l$  and  $\mathbf{z}_i$  be defined as follows:
$$
\mathbf {z} _ {i} = \operatorname {M L P} ^ {(\text {r e l a t i o n})} \left(\left[ \mathbf {h} _ {i} ^ {l}; \mathbf {c} _ {i}; \mathbf {r} _ {i}; \mathbf {u} _ {i}; \mathbf {d} _ {i} ^ {u} \right]\right)
$$
$$
\mathbf {h} _ {i} ^ {l} = \mathrm {L S T M} (\mathbf {h} _ {i} ^ {l - 1}, \mathbf {h} _ {i - 1} ^ {l})
$$
where  $\mathbf{z}_i$  can be thought as a representation of node  $i$  in the graph, conditioned on previous nodes (via  $\mathbf{h}_i^l$  as well as the input text via  $\mathbf{c}_i$ , the graph token (via  $\mathbf{u}_i$  and  $\mathbf{d}_i^u$ ) and the relation type (via  $r_i$ ).
Using this representation  $\mathbf{z}_i$ , Zhang et al. (2019b) introduce an extended pointer-generator network (See et al., 2017) which computes the distribution over the next node label  $v_{i + 1}$ :
$$
\left[ p _ {\text {g e n}}, p _ {\text {e n c}}, p _ {\text {d e c}} \right] = \operatorname {s o f t m a x} \left(\mathrm {M L P} ^ {(\text {s w i t c h})} \left(\mathbf {z} _ {i}\right)\right)
$$
$$
\mathbf {a} _ {i} ^ {\mathrm {d e c}} = \operatorname {s o f t m a x} \left(\operatorname {M L P} ^ {\mathrm {d e c}} ([ \mathbf {z} _ {1: i} ])\right)
$$
$$
\mathbf {p} _ {i} ^ {\left(\text {v o c a b}\right)} = \operatorname {s o f t m a x} \left(\mathrm {M L P} ^ {\left(\text {v o c a b}\right)} \left(\mathbf {z} _ {i}\right)\right)
$$
$$
\mathbf {P} (v _ {i + 1}) = p _ {\mathrm {g e n}} \mathbf {p} _ {i} ^ {(\mathrm {v o c a b})} \oplus p _ {\mathrm {e n c}} \mathbf {a} _ {i} ^ {(\mathrm {e n c})} \oplus p _ {\mathrm {d e c}} \mathbf {a} _ {i} ^ {(\mathrm {d e c})}
$$
From this last equation, we have that the generation of a new node is decomposed into three options: (1) generate a new node from a vocabulary of node labels, (2) copy a node label directly from the input sequence (lexicalization), or (3) copy a node label from a previously generated node (re-entrancy).
Parsing modules To obtain a parse from the node states  $\mathbf{h}_{1:n}$ , a head node and relation type must be assigned to each node  $1:n$ . In order to assign a head node, we instantiate two multi-layer perceptrons (MLPs):  $\mathrm{MLP}^{(\mathrm{start})}$  and  $\mathrm{MLP}^{(\mathrm{end})}$ , where (start) denotes the starting node of the edge and (end) denotes its target. Using these MLPs, for node  $i + 1$  we obtain
$$
\mathbf {h} _ {i + 1} ^ {\text {(s t a r t)}} = \operatorname {M L P} ^ {\text {(s t a r t)}} \left(\mathbf {h} _ {i + 1} ^ {l}\right)
$$
$$
\mathbf {h} _ {1: i} ^ {\text {(e n d)}} = \operatorname {M L P} ^ {\text {(e n d)}} \left(\mathbf {h} _ {1: i} ^ {l}\right)
$$
$$
\mathbf {P} (u _ {i + 1}) = \operatorname {s o f t m a x} \left(\operatorname {B I A F F I N E} \left(\mathbf {h} _ {i + 1} ^ {\text {(s t a r t)}}, \mathbf {h} _ {1: i} ^ {\text {(e n d)}}\right)\right)
$$
The next relationship  $r_{i+1}$  is computed in a similar fashion, also using two MLPs:
$$
\mathbf {h} _ {i + 1} ^ {\left(\text {r e l - s r c}\right)} = \operatorname {M L P} ^ {\left(\text {r e l - s r c}\right)} \left(\mathbf {h} _ {j} ^ {l}\right)
$$
$$
\mathbf {h} _ {i + 1} ^ {(\mathrm {r e l - t g t})} = \operatorname {M L P} ^ {(\mathrm {r e l - t g t})} (\mathbf {h} _ {i + 1} ^ {l})
$$
$$
\mathbf {P} \left(r _ {i + 1}\right) = \operatorname {s o f t m a x} \left(\operatorname {B I L I N E A R} \left(\mathbf {h} _ {i + 1} ^ {\text {(r e l - s r c)}}, \mathbf {h} _ {i + 1} ^ {\text {(r e l - t g t)}}\right)\right)
$$
where  $j$  is the index of the head assigned to the node indexed by  $i + 1$ .<sup>3</sup>
$^{3}$ BIAFFINE is defined in Dozat and Manning (2016). BILINEAR  $(x_{1}, x_{2}) = x_{1}Ax_{2} + b$  where  $A$  and  $b$  are learned parameters.
Node attribute module As noted in previous UDS projects, an important step in decompositional attribute annotation is determining whether a property applies in a given context. For example, factuality typically applies only to predicate nodes. Since all nodes (predicate and argument) are treated identically w.r.t. their semantic relations  $\mathbf{z}_i$ , this work introduces a two-fold node attribute model, which predicts whether a property  $j$  applies to a node  $i$  via a binary mask  $\alpha_i^j$  as well as its value  $\nu_i^j$ . This module defines  $\alpha_i^j$  and  $\nu_i^j$  as follows:
$$
\mathbf {P} \left(\alpha_ {i} ^ {j}\right) = \operatorname {s i g m o i d} \left(\mathrm {M L P} ^ {\left(\text {n o d e - m a s k}\right)} \left(\mathbf {z} _ {i}\right)\right)
$$
$$
\nu_ {i} ^ {j} = \operatorname {M L P} ^ {\left(\text {n o d e - a t t r}\right)} (\mathbf {z} _ {i})
$$
Edge attribute module As in the case of node attributes, edge attributes do not apply in all cases. Therefore, a similar bifurcation strategy is pursued with edge attribute prediction: we predict a binary attribute mask  $\beta_{s,e}^{j}$  for attribute  $j$  on edge  $s\to e$  as well as an attribute value  $\lambda_{s,e}^{j}$ . These are given by:
$$
\mathbf {m} _ {s, e} ^ {(\mathrm {m a s k})} = \operatorname {B I L I N E A R} ^ {(\mathrm {m a s k})} \left(\mathbf {h} _ {s} ^ {l}, \mathbf {h} _ {e} ^ {l}\right)
$$
$$
\mathbf {m} _ {s, e} ^ {\mathrm {(a t t r)}} = \mathrm {B I L I N E A R} ^ {\mathrm {(a t t r)}} (\mathbf {h} _ {s} ^ {l}, \mathbf {h} _ {e} ^ {l})
$$
$$
\mathbf {P} \left(\beta_ {s, e} ^ {j}\right) = \operatorname {s i g m o i d} \left(\operatorname {M L P} ^ {\left(\text {e d g e - m a s k}\right)} \left(\mathbf {m} _ {s, e} ^ {\left(\text {m a s k}\right)}\right)\right)
$$
$$
\lambda_ {s, e} ^ {j} = \operatorname {M L P} ^ {\left(\text {e d g e - a t t r}\right)} \left(\mathbf {m} _ {s, e} ^ {\left(\text {a t t r}\right)}\right)
$$
Training The nodes in the graph are linearized in a pre-order traversal over the arborescence, which ensures that at prediction time, we have seen the potential antecedent of a node for target-side copying (e.g.  $Bush_{(1)}$  in Fig. 2), determining the order of semantic nodes in the graph. The syntactic children of these nodes are presented in the order they appear in the text. The loss functions for the node, head, and relation prediction modules are cross-entropy loss, while for the masks  $\alpha$  and  $\beta$  binary cross-entropy loss is used, since each position in the mask is a separate classification decision. The loss function used for  $K$  attributes  $\nu^{1:K}$  on  $N$  nodes/edges is given by:
$$
\tau (x) = \left\{ \begin{array}{l l} 0 & \text {i f} x \leq 0 \\ 1 & \text {o t h e r w i s e} \end{array} \right.
$$
$$
\mathcal {L} _ {\mathrm {M S E}} (\nu , \nu^ {*}) = \frac {1}{N K} \sum_ {i = 1} ^ {N} \sum_ {j = 1} ^ {K} c _ {i} ^ {j} (\nu_ {i} ^ {j} - \nu_ {i} ^ {j *}) ^ {2}
$$
8431
$$
\begin{array}{l} \mathcal {L} _ {\mathrm {B C E}} (\nu , \nu^ {*}) = \frac {1}{N K} \sum_ {i = 1} ^ {N} \sum_ {j = 1} ^ {K} \left(\tau \left(\nu_ {i} ^ {j *}\right) \log \left(\tau \left(\nu_ {i} ^ {j}\right)\right) \right. \\ \left. + \left(1 - \tau \left(\nu_ {i} ^ {j *}\right)\right) \log \left(1 - \tau \left(\nu_ {i} ^ {j}\right)\right)\right) \\ \end{array}
$$
$$
\mathcal {L} (\nu , \nu^ {*}) = \gamma \frac {2 * \mathcal {L} _ {\mathrm {M S E}} (\nu , \nu^ {*}) * \mathcal {L} _ {\mathrm {B C E}} (\nu , \nu^ {*})}{\mathcal {L} _ {\mathrm {M S E}} (\nu , \nu^ {*}) + \mathcal {L} _ {\mathrm {B C E}} (\nu , \nu^ {*})}
$$