COMBINING GENERATIVE ALIGNERS AND TRANSITION-BASED PARSERS

Systems, computer-implemented methods, and computer program products to facilitate reducing error propagation when combining generative aligners and transition-based aligners are provided. According to an embodiment, a system can comprise a processor that executed components stored in memory. The computer executable components comprise a generative alignment component, an error propagation component, a discriminative parser, and a stochastic oracle policy component. The error propagation component can compute a posterior distribution over one or more hard alignments of parts given a pair of the generative alignment component. The discriminative parser can be trained via the stochastic oracle policy component to reduce error propagation when combining the generative alignment component with the discriminative parser.

The following disclosure is submitted under 35 U.S.C. 102(b)(1)(A):

BACKGROUND

When using a stage of generative alignment to produce data for training a discriminative transition-based parser, error can propagate from inaccuracies on generative alignment to the parser training stage, causing the parser to be less robust. Such inaccuracies can lead to over-fitting to bad alignments. The present disclosure relates to spoken language understanding, and more specifically, to techniques for propagating the uncertainty of the generative stage to the parser training stage using a stochastic oracle policy.

SUMMARY

According to an embodiment, a system can comprise a processor that executes computer executable components stored in memory. The computer executable components comprise a generative alignment component and an error propagation component that can compute a posterior distribution over one or more hard alignments of parts given a pair of the generative alignment component.

According to an embodiment, the computer executable components can comprise a discriminative parser trained via a stochastic oracle policy component to reduce error propagation when combining the generative alignment component with the discriminate parser. Further, in embodiments, the stochastic oracle policy component can perform a learning step on the discriminative parser over one or more epochs. With embodiments, the learning step can comprise learning via gradient descent over one or more epochs.

According to another embodiment, the computer executable components can further comprise a scaling component that can apply a scaling factor to average over the one or more samples. With embodiments, the discriminative parser can be exposed to one or more explanatory models. Additionally, the discriminate parser and the generative alignment component can be end-to-end systems.

An advantage of the above-indicated system can be mitigating the propagation of errors from the aligner training into the parser training. Further, another advantage of the system can be to propagate the uncertainty of the generative stage to the parser training stage.

An additional advantage of the above-indicated system can be that the system provides a regularization effect that prevents overfitting to bad alignments. Further, an advantage of the system can be that the parser is exposed to multiple explanatory models which can assist the model in ignoring bad/improper explanations caused by poorly generalizing rule-based components of the oracle.

According to another embodiment, a computer-implemented method for reducing error propagation when combining generative aligners and transition-based aligners can comprise computing, by a device operatively coupled to a processor, a posterior distribution over one or more hard alignment of parts given a pair within a generative alignment. The computer-implemented method can additionally comprise running, by the device, a stochastic oracle policy to train a discriminative parser. The computer-implemented method can further comprise sampling, by the device, the one or more hard alignments of parts from the posterior distribution. In embodiments, the computer-implemented method can comprise running, by the device, a deterministic oracle policy on one or more samples of the one or more hard alignments of parts. Additionally, the computer-implemented method can comprise performing, by the device, a learning step on the discriminative parser. In embodiments, the computer-implemented method can comprise applying, by the deice, a scaling factor to average over the one or more samples.

According to an embodiment, the computer-implemented method can comprise incorporating an uncertainty of the one or more hard alignments of parts into training the discriminative parser. Additionally, the computer-implemented method can comprise exposing the discriminative parser to one or more explanatory models.

According to another embodiment, running the stochastic oracle policy can prevent over-fitting the discriminate parser to bad alignments. Further, in embodiments, the learning step can be performed via gradient descent. Additionally, the computer-implemented method can comprise transferring embedding-level knowledge from the generative alignment into speech embeddings. With embodiments, the pair can include a natural language portion and a formal representation portion associated with an input sequence.

An advantage of the above-indicated computer-implemented method can be mitigating the propagation of errors from the aligner training into the parser training. Further, another advantage of the system can be to propagate the uncertainty of the generative stage to the parser training stage.

An additional advantage of the above-indicated computer-implemented method can be that the system provides a regularization effect that prevents overfitting to bad alignments. Further, an advantage of the computer-implemented method can be that the parser is exposed to multiple explanatory models which can assist the model in ignoring bad/improper explanations caused by poorly generalizing rule-based components of the oracle.

According to another embodiment, a computer program product for reducing error propagation when combining generative aligners and transition-based aligners, the computer program product comprising a non-transitory computer readable storage memory having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to compute a posterior distribution over one or more hard alignments of parts give a pair within a generative alignment. The computer program product can additionally cause the processor to run a stochastic oracle policy to train a discriminative parser, wherein an uncertainty of alignment of the generative alignment is incorporated into training the discriminative parser.

According to an embodiment, the program instructions can cause the processor to sample the one or more hard alignments of parts from the posterior distribution; run a deterministic oracle policy on one or more samples of the one or more hard alignments of parts; perform a learning step on the discriminative parser; and apply a scaling factor to average over the one or more samples.

According to another embodiment, the program instructions can cause the processor to expose the discriminate parser to one or more explanatory models. With embodiments, the program instructions can cause the processor to prevent over-fitting of the discriminative parser to bad alignments via the stochastic oracle policy. Additionally, the program instructions can cause the processor to perform the learning step via a gradient descent, and can cause the processor to transfer embedding-level knowledge from the generative alignment into speech embeddings.

An advantage of the above-indicated computer program product can be mitigating the propagation of errors from the aligner training into the parser training. Further, another advantage of the computer program product can be to propagate the uncertainty of the generative stage to the parser training stage.

An additional advantage of the above-indicated computer program product can be that the system provides a regularization effect that prevents overfitting to bad alignments. Further, an advantage of the computer program product can be that the parser is exposed to multiple explanatory models which can assist the model in ignoring bad/improper explanations caused by poorly generalizing rule-based components of the oracle.

DETAILED DESCRIPTION

Generally, Abstract Meaning Representation (AMR) can be used as an effort to unify various semantic tasks such as entity-typing, co-reference, relation extraction, and etc. Of existing approaches for AMR parsing, transition-based parsing is notable for high performance while relying on node-to-word alignments as a core pre-processing step. Such alignments are not in the training data and can be learned separately via a complex pipeline of rule-based systems, pre-processing (e.g., lemmatization), and post-processing to satisfy domain-specific constraints. Such pipelines can fail to generalize well, thus propagating errors into training that reduce AMR performance in new domains. Further, alignments can be probabilistically induced and can be used for transition-based AMR parsing in a domain-agnostic manner, which can replace the existing heuristics-based pipeline.

Given problems described above with existing AMR technologies, the present disclosure can be implemented to produce a solution to these problems in the form of systems, computer-implemented methods, and/or computer program products that can facilitate mitigating the propagation of errors from the aligner training into the parser training by: training the AMR parser on oracle actions derived samples from the neural aligner's (e.g., the generative aligner or generative alignment component102) posterior distribution. The neural aligner can be used as an importance sampling distribution, which can be used to better approximate samples from the AMR parser's posterior alignment distribution, and thus can better approximate the otherwise intractable log marginal likelihood.

An advantage of the above-indicated system can be a simplified pipeline that can learn state-of-the-art AMR parsers that perform well on both AMR2.0 and AMR3.0. AMR parsers learned in this manner do not require beam search and hence can be more efficient at test time. Additionally, another advantage of the above-indicated system can be to transfer strong inductive biases from the aligner (e.g., the constrained aligner) to the parser (e.g., the overly flexible parser).

Given these problems, one or more embodiments described herein can be implemented to produce a solution to one or more of these problems in the form of systems, computer-implemented methods, and/or computer program products that can facilitate the following processes: i) computing a posterior distribution over one or more hard alignments of parts given a pair within a generative alignment; ii) running a deterministic oracle policy on the one or more samples of the one or more hard alignments of parts; iii) sampling the one or more hard alignments of parts from the posterior distribution; iv) running a deterministic oracle policy on one or more samples of the one or more hard alignments of parts; v) performing a learning step on the discriminative parser; and vi) applying a scaling factor to average over the one or more samples. That is, embodiments described herein include one or more systems, computer implemented methods, apparatuses and/or computer program products that can facilitate one or more of the aforementioned processes.

FIG.1illustrates a block diagram of an example, non-limiting error mitigation system100that comprises a generative alignment component102(e.g., an end-to-end system), an error propagation component104, a discriminative parser106(e.g., an end-to-end system), a stochastic oracle policy component108, and a scaling component110. Additionally, the error propagation component104can compute a posterior distribution over one or more hard alignments of parts given a pair of the generative alignment component102. Repetitive description of like elements employed in other embodiments described herein is omitted for sake of brevity. Aspects of systems (e.g., the error mitigation system100and the like), apparatuses or processes in various embodiments of the present invention can constitute one or more machine-executable components embodied within one or more machines (e.g., embodied in one or more computer readable mediums (or media) associated with one or more machines). Such components, when executed by the one or more machines (e.g., computers, computing devices, virtual machines, a combination thereof, and/or the like) can cause the machines to perform the operations described.

Additional description of functionalities will be further described below with reference to the example embodiments ofFIG.1, where repetitive description of like elements and/or processes employed in respective embodiments is omitted for sake of brevity. The error mitigation system100can facilitate: i) computing a posterior distribution over one or more hard alignments of parts given a pair within a generative alignment; ii) running a deterministic oracle policy on the one or more samples of the one or more hard alignments of parts; iii) sampling the one or more hard alignments of parts from the posterior distribution; iv) running a deterministic oracle policy on one or more samples of the one or more hard alignments of parts; v) performing a learning step on the discriminative parser; and vi) applying a scaling factor to average over the one or more samples. The generative alignment component102, the error propagation component104, the discriminative parser106, the stochastic oracle policy component108, and scaling component110can be associated with a computing environment600(FIG.6).

Discussion first turns briefly to system bus120, processor122, and memory124of error mitigation system100. For example, in one or more embodiments, the error mitigation system100can comprise processor122(e.g., computer processing unit, microprocessor, classical processor, and/or like processor). In one or more embodiments, a component associated with error mitigation system100, as described herein with or without reference to the one or more figures of the one or more embodiments, can comprise one or more computer and/or machine readable, writable and/or executable components and/or instructions that can be executed by processor122to enable performance of one or more processes defined by such component(s) and/or instruction(s).

In one or more embodiments, error mitigation system100can comprise a computer-readable memory (e.g., memory124) that can be operably connected to the processor122. Memory124can store computer-executable instructions that, upon execution by processor122, can cause processor122and/or one or more other components of the error mitigation system100(e.g., generative alignment component102, error propagation component104, discriminative parser106, stochastic oracle policy component108, and scaling component110) to perform one or more actions. In one or more embodiments, memory124can store computer-executable components (e.g., generative alignment component102, error propagation component104, discriminative parser106, stochastic oracle policy component108, and scaling component110).

With embodiments, error mitigation system100and/or a component thereof as described herein, can be communicatively, electrically, operatively, optically and/or otherwise coupled to one another via bus120. Bus120can comprise one or more of a memory bus, memory controller, peripheral bus, external bus, local bus, and/or another type of bus that can employ one or more bus architectures. One or more of these examples of bus120can be employed. In one or more embodiments, the error mitigation system100can be coupled (e.g., communicatively, electrically, operatively, optically and/or like function) to one or more external systems (e.g., a non-illustrated electrical output production system, one or more output targets, an output target controller and/or the like), sources and/or devices (e.g., classical computing devices, communication devices and/or like devices), such as via a network. In one or more embodiments, one or more of the components of the error mitigation system100can reside in the cloud, and/or can reside locally in a local computing environment (e.g., at a specified location(s)). In examples, the error mitigation system100can be connected with the bus120, one or more input devices132, and one or more computer applications134, which can be associated with cloud computing environment600(FIG.6).

In addition to the processor122and/or memory124described above, the error mitigation system100can comprise one or more computer and/or machine readable, writable and/or executable components and/or instructions that, when executed by processor122, can enable performance of one or more operations defined by such component(s) and/or instruction(s). The error mitigation system100can be associated with, such as accessible via, a computing environment600described below with reference toFIG.6. For example, error mitigation system100can be associated with a computing environment600such that aspects of processing can be distributed between the error mitigation system100and the computing environment600.

With embodiments, such as generally illustrated byFIGS.1and2, an effective way to train AMR parsers can be with sequence-to-sequence learning where the input sequence can be the sentence w (e.g., seeFIG.3) and the output sequence can be a graph g (see, e.g.,FIG.3) decomposed into an action sequence a via an oracle (e.g., the stochastic oracle policy component108). An example input sentence w is illustrated inFIG.3, that includes the phrase “The harder they come, the harder they fall.” The combination of words (e.g., text) and actions (e.g., structure) can be provided to a parameter-less state machine M that can produce the graph g:=M(w, a). The state machine can perform the oracle inverse operation O when also provided alignments I, mapping a graph to a deterministic sequence of oracle actions a:=O(l, w, g). During training, the model can learn to map w→a (e.g., these pairs are given by the oracle O) and M can be used to construct graphs (a→g) for evaluation.

In embodiments, such as generally illustrated inFIG.2, the system100can comprise the alignment model (e.g., generative alignment component102) which can be trained to maximize the probability of the formal representation given the input natural language sentence w. The sentence w can include a natural language pairs that can be split into text and structure portions. Further, the alignment distribution can be strongly factored for tractable learning and can omit elements of the formal language that are not explicitly elated to the natural language. For example, the formal representation can be a graph g where word-node pairs and the alignment distribution can be defined to align the word in the input to one or more possible nodes. The graph g can include edges (e.g., seeFIG.3) that need not be aligned if such edges cannot be clearly related to surface symbols. Additionally, training of the alignment distribution can result in the identification of a number of frequent parts and in turn alignments between parts of the natural language and the formal representation. The sentence w can be processed left-to-right as a set of tokens; and, for each node300, aligned nodes can be generated, as well as arcs302between current generated nodes and past generated nodes when applicable (which can be determined by an expert or obtained via combinational search).

With embodiments, given each training pair (e.g., natural language, formal representation) the hard alignment of parts (alignment sample-1 . . . sample-N) can be extracted from the alignment distribution, such as illustrated inFIG.2. A deterministic oracle policy can be applied, via the stochastic oracle policy component108, to the pair (e.g., the text and structure pair/natural language and structure pair) and the hard alignment of parts, providing a set of actions that transform the natural language sentence w into the formal representation.

In embodiments, the oracle and the state machine from StructBART can be used. The oracle and the state machine can rely on rules that can determine which actions are valid (e.g., the first action may not be to generate an edge). Further, the actions can be the output space that the parser predicts, and when read from left-to-right can be used to construct an AMR graph (e.g., such as illustrated by g inFIG.3). The actions can incorporate alignments.

With embodiments, the following rules can define the valid actions at each time step: i) maintain a cursor that reads the sentence left-to-right and progressing for SHIFT action; ii) at each cursor position, generate any nodes aligned to the cursor's word (e.g., node-word alignments can be used); and iii) immediately after generated a node, also generate any valid incoming or outgoing arcs. Further, in additional embodiments, the following actions can be performed at each time step: i) SHIFT: Increment the cursor position; ii) NODE(yv): Generate node with label yv; iii) COPY: Generate a node by copying the word under the cursor as the label; iv) LA(ye, n), RA((ye, n): Generate an edge with label yefrom the most recently generated node to the previously generated node n. LA and RA (e.g., left-arc and right-arc) can indicate the edge direction as outgoing/incoming. yecan be used to different edge labels from node labels yv; and v) END: a special action that can indicate that the full graph has been generated.

In additional embodiments, for parsing, StructBART can fine-tune (Bidirectional and Auto-Regressive Transformer) BART with the following modifications: i) StructBART can convert an attention head from a BART decoder into a pointer network for predicting n in the LA/RA actions; ii) logits for actions can be masked to guarantee graph well-formedness; and iii) alignment can be used to mask two cross-attention heads of the BART decoder, thereby integrating structural alignment directly in the model. Further, StructBART can be trained to optimize the maximum likelihood of action sequences given sentence and alignment. Moreover, for a single example (w, g, l)˜D, a:=O(l, w, g), the log-likelihood of the actions (and the associate graph) can be given by, log p (a|w; θ)=Σt=1Tlog p(at|at<1,w; θ) for a model with parameters θ. Probabilities of actions that create arcs can be decomposed into independent label and pointer distributions:

p (ye|at<1, w; θ)p(n|at<1, w; θ) where can be computed with the normal output vocabulary distribution of BART and with an attention head of the decoder.

With embodiments, for training, StructBART depends on node-to-word AMR alignment l to specify the oracle actions. The steps in SB-Align can comprise: i) producing initial alignment using Symmetrized Expectation Maximization; ii) attempting to align additional nodes by inheriting child node alignments; and iii) continuing to re-fine alignments using JAMR which involves constraint optimization using a set of linguistically motivated rules. The StructBART action space can that all nodes are aligned, however, after running SB-Align some node may not be. This can be resolved by first “force aligning” unaligned nodes to unaligned tokes, then propagating alignments from child-to-parent nodes and vice versa until all nodes are aligned to text spans. Node-to-span alignments can be converted into node-to-token alignments for model training (e.g., by deterministically aligning to the first node of an entity).

In embodiments, the neural aligner (e.g., the generative alignment component102) can be a variant of sequence-to-sequence models with hard attention. In contrast to SB-Align, the error mitigation system100can include minimal pre-processing and does not include dependencies on many components or domain-specific rules. The alignment model (e.g., the generative alignment component102) can be trained separately from the AMR parser and can optimize the conditional likelihood of nodes in the linearized graph given the sentence w. Such as generally illustrated inFIG.4A, the AMR graph can be linearized by first converting the graph into a tree (e.g., comprising nodes200and edges202) and then linearizing the tree via a depth-first search. Further, the generative alignment component102can incur a loss for nodes200(e.g., as indicated in bold font inFIG.3andFIG.4A). As seen inFIG.4B, the shaded regions represent the alignment posterior400, and the squares represent a point estimate from the baseline402, which corresponds to the most probable value of each distribution over tokens. Letting v=v1, . . . , vsbe the nodes in the linearized graph, the log-likelihood can be given by:

where we use v<sto indicate all the tokens (e.g., including brackets and edges) before vs. That is, we can incur losses only on the nodes vs, but still can represent the entire history v<sfor the prediction. The probability of each node300can be given by marginalizing over latent alignments ls,

where ls=i indicates that node vscan be aligned to word wi. For parameterization, the sentence w can be encoded by a bi-directional LSTM. Words can be represented using a word embedding derived from a pre-trained character-encoder from ELMo, which can be frozen during training. As illustrated inFIG.4B, the linearized AMR tree can be transformed into a visualization of the alignment posterior and point estimates from the baseline. One the decoder side, the linearized AMR tree history can be represented by a uni-directional LSTM. The decoder can share word embeddings with the text encoder. The prior alignment probability q(ls=i|v<s, w; φ) can be given by bilinear attention:

where W can be a learned matrix, hi(W)can be a concatenation of forward and backward LSTM vectors for the i-th word in the text encoder, and ht(v)can be a vector immediately before the s-th node in the graph decoder. The likelihood q(vs|ls=i, v<s, w; φ) can be formulated as a softmax layer with the relevant vectors concatenated as input,

where the softmax can be over the node vocabulary and can be indexed by the label y belonging to the node vs.

Additionally, once trained, the posterior distribution can be tractably obtained over each alignment ls,

and the full posterior distribution over all alignments l=l1, . . . , lscan be given by q(l|w, g; φ)=Πs=1sq(ls|w, v; φ). With embodiments, as compared to classic count-based alignment models, the neural parametrization can make it easy to utilize pretrained embeddings and also condition on the alignment and emission distribution on richer context. For example and without limitation, the emission distribution q(vsls, v<s, w; φ) can condition on the full target history v<sand the source context w, unlike count-based models which typically condition on just the aligned word wls. The flexible modeling capabilities enabled by the use of neural networks can be useful in obtaining good alignment performance.

In embodiments, the neural aligner (e.g., the generative alignment component102) can induce a posterior distribution over alignments, q(l|w, g; φ). To use the alignment model, the MAP alignment can be decoded {circumflex over (l)}=argmaxlq(l|w, g; φ) and trained from the actions â=O({circumflex over (l)}, w, g). Further, the action sequences derived from MAP alignments do not take into account the uncertainty associated with posterior alignments, which may not be ideal. The AMR parser's posterior can be regularized to be close to the neural aligner's posterior at the distribution level to take the above referenced uncertainty into account.

With embodiments, the action oracle O(l, w, g) can be bijective as a function of l (e.g., keeping w and g fixed), so the transition-based parser p(a|w; θ) induces a joint distribution over alignments and graphs,

This joint distribution can further induce a marginal distribution over graphs,

as well as a posterior distribution over alignments,

In examples, the neural aligner's distribution (e.g., the generative alignment component102) can be used via a posterior regularized likelihood,

Further, the parser can be learned such that it gives a high likelihood to the gold graph g given the sentence w but at the same time has a posterior alignment distribution that is close to the neural aligner's posterior. Rearranging terms can yield,

and since the second term is a constant with respect to θ, the gradient with respect to θ can be given by,

The above Monte Carlo gradient estimator can generalize the MAP alignment case. Further, setting q(l|w, g)={=l=1} where l is the MAP alignment (e.g., or an alignment derived from the existing pipeline) recovers the existing baseline.

With embodiments, the posterior regularized likelihood lower bounds the log marginal likelihood(θ)=log p(g|w; θ) and can implicitly assume that training against the lower bound results in a model that generalizes better than a model trained against the true log marginal likelihood. Additionally, the neural aligner (e.g., the generative alignment component102) can be used as a surrogate posterior distribution whose samples can be reweighted (e.g., via the scaling component110) to reduce the gap between the(θ) and thePR(θ). The product of the neural aligner's posterior can be used to create a joint posterior distribution where K is the number of importance samples:

Further, a scaling factor (1/K) can be applied, via the scaling component110, to perform an arithmetic average. The scaling factor can be equal to the number of alignments drawn from the distribution. Thus, the following objective:

monotonically converges to the log marginal likelihood log p(g|w; θ) as K→∞. A single sample Monte Carlo gradient estimator (e.g., via gradient descent) for the above can be given by:

where further provided are the normalized importance weights, which can be performed via a learning step:

Thus, in embodiments, compared to the gradient estimator in the posterior regularized case which equally weights each sample, the importance-weighted objective approximates the true posterior p(l|w, g; θ) by first sampling from a fixed distribution q(l|w, g; φ and then reweighing it accordingly.

With embodiments, the error mitigation system100differs from the variational approaches in that q can be fixed to the pretrained aligner posterior and it is not optimized further. Moreover, the lower boundPR(θ) can represent an inductive bias informed by a pretrained aligned, which can be more suited for early stages of training than tangent evidence lower bound (e.g., with zero gap). For a tangent lower bound, q in the Monte Carlo gradient estimate (e.g., by gradient descent) can be equal to the true posterior over alignments for the error mitigation system100. Posterior regularization can seek to transfer the neural aligner's strong inductive biases to the AMR parser, which can have weaker inductive biases and thus can be potentially too flexible of a model. On the other hand, importance sampling trusts the AMR parser's inductive bias more, and can use the neural aligner as a surrogate distribution that can be adapted to more closely approximate the AMR parser's intractable posterior. In examples, if the posterior regularized variant outperforms the importance sampling variant, it can suggest that the StructBART is too flexible of a model.

In embodiments, the models can be evaluated on two datasets for AMR parsing in English. AMR2.0 contains about ˜39 k sentences from multiple genres (LDC2017T10). AMR3.0 is a subset of AMR2.0 sentences with approx. 20 k new sentences (LDC2020T02), improved annotations with new fames, annotation corrections, and expanded annotation guidelines. Using AMR3.0 for evaluation can allow measuring of how well the alignment procedure generalizes to new datasets. Further, AMR3.0 can include new sentences but also new genres such as text from LORELEI, Aesop fables, and Wikipedia. The primary evaluation of the aligner is extrinsically through ARM parsing, and we additionally evaluate alignments directly against ground truth annotations. Moreover, the 130 sentences from the ARM2.0 train data (e.g., the most suited for SB-Align) can be examined, which can be labeled the gold test set. Alignment annotation are used for evaluation, and further, they are not used for aligner training.

With embodiments, the text tokens can be aligned to AMR nodes. As the AMR sentences do not include defector tokenization, the strings can be split on space and punctuation using one or more regex rules. For AMR parsing, the action set previously described in this application can be used. To accommodate the recurrent nature of the aligner, the AMR graph can be linearized during aligner training. Such conversion involves converting the graph into a tree and removing re-entrant edges.

Further, for AMR parsing, Smatch can be utilized. Moreover, for AMR alignment the goal can be to compare the new aligner with strong alignment baselines: SB-Align and LEAMR, a state-of-the-art alignment model. However, the aligner can predicts node-to-word alignments, SB-Align can predict node-to-span alignments, and the ground truth alignments can be subgraph-to-span. The mismatch in granularity can be addressed by measuring alignment performance using a permissive version of F1 after decomposing subgraph-to-span alignments into node-to-span alignments (e.g., a prediction is correct if it overlaps with the gold span). Such permissiveness can give advantages to the LEAMR and SB-Align baselines (which can predict span-based alignments) as there is no precision-related penalty for predicting large spans.

With examples, a bi-directional LSTM can be used for the text encoder and a uni-directional LSTM can be used for the AMR decoder. The input token embeddings can be derived from a pretrained character encoder and can be frozen throughout training. Such token embeddings can be tied with the output softmax, which can allow for alignment tokens not seen during training. The alignment model can otherwise be parameterized. Training can include a duration of about 200 epochs. Further, training can be unsupervised so that final checkpoint can be used.

In additional embodiments, the StructBART model can be used to fine-tuning for 100 epochs (AMR2.0) or 120 epochs (AMR3.0), and additionally Smatch can be used on the validation set for early stopping. For example, the hyperparameters of the AMR parser are not tuned so that it can be evaluated on how well the neural aligner (e.g., the generative alignment component102) performed as a “plug-in” to an existing system. Experimental results on parsing for AMR2.0 and AMR3.0 test sets are reported in Table 1 wherein numbers are reported when using single alignments (MAP), posterior regularization (PR), and importance sampling (IS). Additionally, Table 1 includes a number of silver data training sentences used and the corresponding beam size. As can be seen from Table 1, PR and IS do not improve with beam search (numbers are omitted). The nomenclature “P” indicates using partial ensemble for the decoder, and “G” indicates using graph recategorization.

With embodiments, SB-Align was developed prior to the AMR3.0 release, and because it incorporates a complex pipeline with domain-specific rules, it can be considered specialized for prior datasets like AMR2.0. As shown in Table 1, the aligner yields relatively stronger StructBART improvements for AMR3.0 than AMR2.0. This result and the relatively little manual configuration for the aligner (e.g., no rules, lemmatization, etc.) suggest the alignment approach generalizes better to different training corpora and that prior performance of StructBART on AMR3.0 can be affected by a lack of generalization. Graph re-categorization can be used in AMR parsing where groups of nodes are collapsed during training and test time but expanded during evaluation. The categorization can be harmful, by the results indicate that re-categorization is partially a function of alignment-like heuristics and the lower re-categorization results of SPRING in AMR3.0 reinforce the findings that alignments based on heuristic can be difficult to generalize.

In embodiments, the parsing performance can be improved by sampling5alignment per sentence in batch (see, e.g., Table 1). When looking at the sampling results compared with previous versions of StructBART trained on silver data, the error mitigation system100can outperform the benefit of simpler versions of data augmentation, such as simple self-learning.

Additionally, in further embodiments, training with posterior regularization or importance sampling uses the same number of samples, but in different ways. For example, in posterior regularization, the samples can be used to better approximate the posterior regularized objective, which in turn can regularize the AMR parser's posterior more effectively which can reduce the gradient estimator variance. Further, in importance sampling, the samples can be used to better approximate the AMR parser's intractable log marginal likelihood. The importance sampling fails to improve upon posterior regularization for both AMR2.0 and AMR3.0, which can indicate that strong inductive biases associated the constrained aligner is a useful training signal for the flexible AMR parser.

The neural alignment method can be preferred over SB-Align due to relatively easy use (e.g., makes use of word embeddings, depends on less preprocessing, does not require domain specific rules, etc.) and can empirically improve performance (see, e.g., Table 1). To verify that improved parsing is due to better alignment, a comparison can be drawn against two strong alignment baselines (LEAMR and SB-Align) on an evaluation set of gold manually annotated alignments. Generally, a few hundred annotations can be available, and the goal can be to use such alignments on 10s or 100s of thousands of sentences for AMR parsing. For this reason, the aligners can be trained unsupervised with respect to alignment. Table 2 demonstrates results that show the neural aligner performs substantially better than SB-Align and the neural aligner is nearly performing the same as LEAMR (e.g., the current state of the art).

In embodiments, the count-based IBM model1can be trained using expectation maximization. Further, the neural aligner can be trained without pretrained character-aware embeddings. The neural aligner can learn the prior alignment distribution and the emission model conditions on the entire sentence w and the target history v<s. Pretrained embeddings can be added to the model to recover the full model. The results in Table 2 indicate both flexibility and token representation outperform IBM Model1. In examples, training with word vectors learned from scratch provides a small benefit compares to using pretrained character embeddings, which can yield about 20 point improvement in the permissive F1 metric.

FIG.5illustrates a flow diagram of an example, non-limiting computer-implemented method500that can facilitate mitigating the propagation of errors from the aligner training stage to the parser training in accordance with one or more embodiments described herein. Repetitive description of like elements and/or processes employed in respective embodiments is omitted for sake of brevity.

At502, the computer-implemented method500can comprise computing, by a device operatively coupled to the processor122(e.g., the generative alignment component102, the error propagation component104, discriminative parser106, stochastic oracle policy component108, and/or the scaling component110), a posterior distribution over one or more hard alignments of parts given a pair within a generative alignment.

At504, the computer-implemented method500can comprise running, by the device (e.g., the generative alignment component102, the error propagation component104, discriminative parser106, stochastic oracle policy component108, and/or the scaling component110), a stochastic oracle policy to train a discriminative parser106.

At506, the computer-implemented method500, and/or step504can comprise sampling, by the device (e.g., the generative alignment component102, the error propagation component104, discriminative parser106, stochastic oracle policy component108, and/or the scaling component110), the one or more hard alignments of parts from the posterior distribution.

At508, the computer-implemented method500, and/or step504can comprise running, by the device (e.g., the generative alignment component102, the error propagation component104, discriminative parser106, stochastic oracle policy component108, and/or the scaling component110), a deterministic oracle policy on one or more samples of the one or more hard alignment of parts.

At510, the computer-implemented method500, and/or step504can comprise performing, by the device (e.g., the generative alignment component102, the error propagation component104, discriminative parser106, stochastic oracle policy component108, and/or the scaling component110), a learning step on the discriminative parser106.

At512, the computer-implemented method500, and/or step504can comprise applying, by the device (e.g., the generative alignment component102, the error propagation component104, discriminative parser106, stochastic oracle policy component108, and/or the scaling component110), a scaling factor to average over the one or more samples. Further, the method500can additionally comprise incorporating an uncertainty of the one or more hard alignments of parts into training the discriminative parser106. In embodiment, the method500can comprise exposing the discriminative parser106to bad alignments, and performing the learning step via a gradient descent.

For simplicity of explanation, the computer-implemented and non-computer-implemented methodologies provided herein are depicted and/or described as a series of acts. It is to be understood that the subject innovation is not limited by the acts illustrated and/or by the order of acts, for example acts can occur in one or more orders and/or concurrently, and with other acts not presented and described herein. Furthermore, not all illustrated acts can be utilized to implement the computer-implemented and non-computer-implemented methodologies in accordance with the described subject matter. Additionally, the computer-implemented methodologies described hereinafter and throughout this specification are capable of being stored on an article of manufacture to enable transporting and transferring the computer-implemented methodologies to computers. The term article of manufacture, as used herein, is intended to encompass a computer program accessible from any computer-readable device or storage media.

The systems and/or devices have been (and/or will be further) described herein with respect to interaction between one or more components. Such systems and/or components can include those components or sub-components specified therein, one or more of the specified components and/or sub-components, and/or additional components. Sub-components can be implemented as components communicatively coupled to other components rather than included within parent components. One or more components and/or sub-components can be combined into a single component providing aggregate functionality. The components can interact with one or more other components not specifically described herein for the sake of brevity, but known by those of skill in the art.

One or more embodiments described herein can employ hardware and/or software to solve problems that are highly technical, that are not abstract, and that cannot be performed as a set of mental acts by a human. For example, a human, or even thousands of humans, cannot efficiently, accurately and/or effectively mitigate the prorogation of errors from the generative stage to the parser training stage as the one or more embodiments described herein can enable this process. And, neither can the human mind nor a human with pen and paper mitigate the propagation of errors from the generative stage to the parser training stage, as conducted by one or more embodiments described herein.

FIG.6illustrates a block diagram of an example, non-limiting operating environment600in which one or more embodiments described herein can be facilitated.FIG.6and the following discussion are intended to provide a general description of a suitable operating environment600in which one or more embodiments described herein atFIGS.1-5can be implemented.

Computing environment600contains an example of an environment for the execution of at least some of the computer code involved in performing the inventive methods, such as neural aligner code645. In addition to block645, computing environment600includes, for example, computer601, wide area network (WAN)602, end user device (EUD)603, remote server604, public cloud605, and private cloud606. In this embodiment, computer601includes processor set610(including processing circuitry620and cache621), communication fabric611, volatile memory612, persistent storage613(including operating system622and block645, as identified above), peripheral device set614(including user interface (UI), device set623, storage624, and Internet of Things (IoT) sensor set625), and network module615. Remote server604includes remote database630. Public cloud605includes gateway640, cloud orchestration module641, host physical machine set642, virtual machine set643, and container set644.

PROCESSOR SET610includes one, or more, computer processors of any type now known or to be developed in the future. Processing circuitry620may be distributed over multiple packages, for example, multiple, coordinated integrated circuit chips. Processing circuitry620may implement multiple processor threads and/or multiple processor cores. Cache621is memory that is located in the processor chip package(s) and is typically used for data or code that should be available for rapid access by the threads or cores running on processor set610. Cache memories are typically organized into multiple levels depending upon relative proximity to the processing circuitry. Alternatively, some, or all, of the cache for the processor set may be located “off chip.” In some computing environments, processor set610may be designed for working with qubits and performing quantum computing.

VOLATILE MEMORY612is any type of volatile memory now known or to be developed in the future. Examples include dynamic type random access memory (RAM) or static type RAM. Typically, the volatile memory is characterized by random access, but this is not required unless affirmatively indicated. In computer601, the volatile memory612is located in a single package and is internal to computer601, but, alternatively or additionally, the volatile memory may be distributed over multiple packages and/or located externally with respect to computer601.

END USER DEVICE (EUD)603is any computer system that is used and controlled by an end user (for example, a customer of an enterprise that operates computer601), and may take any of the forms discussed above in connection with computer601. EUD603typically receives helpful and useful data from the operations of computer601. For example, in a hypothetical case where computer601is designed to provide a recommendation to an end user, this recommendation would typically be communicated from network module615of computer601through WAN602to EUD603. In this way, EUD603can display, or otherwise present, the recommendation to an end user. In some embodiments, EUD603may be a client device, such as thin client, heavy client, mainframe computer, desktop computer and so on.

REMOTE SERVER604is any computer system that serves at least some data and/or functionality to computer601. Remote server604may be controlled and used by the same entity that operates computer601. Remote server604represents the machine(s) that collect and store helpful and useful data for use by other computers, such as computer601. For example, in a hypothetical case where computer601is designed and programmed to provide a recommendation based on historical data, then this historical data may be provided to computer601from remote database630of remote server604.

PRIVATE CLOUD606is similar to public cloud605, except that the computing resources are only available for use by a single enterprise. While private cloud606is depicted as being in communication with WAN602, in other embodiments a private cloud may be disconnected from the internet entirely and only accessible through a local/private network. A hybrid cloud is a composition of multiple clouds of different types (for example, private, community or public cloud types), often respectively implemented by different vendors. Each of the multiple clouds remains a separate and discrete entity, but the larger hybrid cloud architecture is bound together by standardized or proprietary technology that enables orchestration, management, and/or data/application portability between the multiple constituent clouds. In this embodiment, public cloud605and private cloud606are both part of a larger hybrid cloud.

Aspects of the one or more embodiments described herein are described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to one or more embodiments described herein. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions. These computer readable program instructions can be provided to a processor of a general-purpose computer, special purpose computer and/or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, can create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus and/or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein can comprise an article of manufacture including instructions which can implement aspects of the function/act specified in the flowchart and/or block diagram block or blocks. The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus and/or other device to cause a series of operational acts to be performed on the computer, other programmable apparatus and/or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable apparatus and/or other device implement the functions/acts specified in the flowchart and/or block diagram block or blocks.

The flowcharts and block diagrams in the figures illustrate the architecture, functionality and/or operation of possible implementations of systems, computer-implementable methods and/or computer program products according to one or more embodiments described herein. In this regard, each block in the flowchart or block diagrams can represent a module, segment and/or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function. In one or more alternative implementations, the functions noted in the blocks can occur out of the order noted in the Figures. For example, two blocks shown in succession can be executed substantially concurrently, and/or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and/or combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that can perform the specified functions and/or acts and/or carry out one or more combinations of special purpose hardware and/or computer instructions.

While the subject matter has been described above in the general context of computer-executable instructions of a computer program product that runs on a computer and/or computers, those skilled in the art will recognize that the one or more embodiments herein also can be implemented at least partially in parallel with one or more other program modules. Generally, program modules include routines, programs, components and/or data structures that perform particular tasks and/or implement particular abstract data types. Moreover, the aforedescribed computer-implemented methods can be practiced with other computer system configurations, including single-processor and/or multiprocessor computer systems, mini-computing devices, mainframe computers, as well as computers, hand-held computing devices (e.g., PDA, phone), and/or microprocessor-based or programmable consumer and/or industrial electronics. The illustrated aspects can also be practiced in distributed computing environments in which tasks are performed by remote processing devices that are linked through a communications network. However, one or more, if not all aspects of the one or more embodiments described herein can be practiced on stand-alone computers. In a distributed computing environment, program modules can be located in both local and remote memory storage devices.