Introduction
Accurate and efficient semantic parsing is a long-standing goal in natural language processing. There are countless applications for methods that provide deep semantic analyses of sentences. Leveraging semantic information in text may provide improved algorithms for many problems in NLP, such as named entity recognition BIBREF0 , BIBREF1 , BIBREF2 , word sense disambiguation BIBREF3 , BIBREF4 , semantic role labeling BIBREF5 , co-reference resolution BIBREF6 , BIBREF7 , etc. A sufficiently expressive semantic parser may directly provide the solutions to many of these problems. Lower-level language processing tasks, such as those mentioned, may even benefit by incorporating semantic information, especially if the task can be solved jointly during semantic parsing.
Knowledge plays a critical role in natural language understanding. The formalisms used by most semantic parsing approaches require an ontology of entities and predicates, with which the semantic content of sentences can be represented. Moreover, even seemingly trivial sentences may have a large number of ambiguous interpretations. Consider the sentence “She started the machine with the GPU,” for example. Without additional knowledge, such as the fact that “machine” can refer to computing devices that contain GPUs, or that computers generally contain devices such as GPUs, the reader cannot determine whether the GPU is part of the machine or if the GPU is a device that is used to start machines.
The thesis underlying our research is that natural language understanding requires a belief system; that is, a large set of pre-existing beliefs related to the domain of discourse. Clearly, young children have many beliefs about the world when they learn language, and in fact, the process of learning language is largely one of learning to ground the meanings of words and sentences in these non-linguistically acquired beliefs. In some ways, the idea that language understanding requires a belief system is not new, as natural language researchers have been saying for years that background knowledge is essential to reducing ambiguity in sentence meanings BIBREF8 , BIBREF9 , BIBREF10 , BIBREF11 . But despite this general acknowledgement of the importance of background knowledge, we see very few natural language understanding systems that actually employ a large belief system as the basis for comprehending sentence meanings, and for determining whether the meaning of a new sentence contradicts, extends, or is already present in its belief system.
We present here a step in this direction: a probabilistic semantic parser that uses a large knowledge base (NELL) to form a prior probability distribution on the meanings of sentences it parses, and that "understands" each sentence either by identifying its existing beliefs that correspond to the sentence's meaning, or by creating new beliefs. More precisely, our semantic parser corresponds to a probabilistic generative model that assigns high probability to sentence semantic parses resulting in beliefs it already holds, lower prior probability to parses resulting in beliefs it does not hold but which are consistent with its more abstract knowledge about semantic types of arguments to different relations, and still lower prior probability to parses that contradict its beliefs about which entity types can participate in which relations.
This work is only a first step. It is limited in that we currently use it to parse sentences with a simple noun-verb-noun syntax (e.g. "Horses eat hay."), and considers only factual assertions in declarative sentences. Its importance is that it introduces a novel approach in which the semantic parser (a) prefers sentence semantic parses that yield assertions it already believes, while (b) still allowing with lower prior probability sentence interpretations that yield new beliefs involving novel words, and (c) even allowing beliefs inconsistent with its background knowledge about semantic typing of different relations. We introduce algorithms for training the probabilistic grammar and producing parses with high posterior probability, given its prior beliefs and a new sentence. We present experimental evidence of the success and tractability of this approach for sentences with simple syntax, and evidence showing that the incorporated belief system, containing millions of beliefs, allows it to outperform state-of-the-art semantic parsers that do not hold such beliefs. Thus, we provide a principled, probabilistic approach to using a current belief system to guide semantic interpretation of new sentences which, in turn, can be used to augment and extend the belief system. We also argue that our approach can be extended to use the document-level context of a sentence as an additional source of background beliefs.
For reasons including but not limited to performance and complexity, most modern parsers operate over tokens, such as words. While this has worked sufficiently well for many applications, this approach assumes that a tokenization preprocessing step produces the correct output. This is nontrivial in many languages, such as Chinese, Thai, Japanese, and Tibetic languages. In addition, a large portion of the English vocabulary is created from the combination of simpler morphemes, such as the words “build-er,” “in-describ-able,” “anti-modern-ist.” Moreover, language can be very noisy. Text messages, communication in social media, and real-world speech are but a few examples of noise obfuscating language. Standard algorithms for tokenization, lemmatization, and other preprocessing are oblivious to the underlying semantics, much less any background knowledge. Incorporating these components into a “joint parsing” framework will enable semantics and background knowledge to jointly inform lower-level processing of language. Our method couples semantics with syntax and other lower-level aspects of language, and can be guided by background knowledge via the semantic prior. We will demonstrate how this can be leveraged in our framework to model the morphology of individual verbs in a temporally-scoped relation extraction task.
Semantic statements are the logical expressions that represent meaning in sentences. For example, the semantic statement turn_on_device(person:Ada, device:gpu_cluster) may be used to express the meaning of the sentence example given earlier. There are many languages or semantic formalisms that can be used to encode these logical forms: first-order logic with lambda calculus BIBREF12 , frame semantics BIBREF13 , abstract meaning representation BIBREF14 , dependency-based compositional semantics BIBREF15 , vector-space semantics BIBREF16 , BIBREF17 , for example. Our approach is flexible and does not require the use of a specific semantic formalism.
In section "Hierarchical Dirichlet processes" , we review HDPs and describe the setting that we require to define our grammar. We present our approach in section UID17 to perform HDP inference in this new setting. In section "Generative semantic grammar" , we present the main generative process in our framework, and detail our application of the HDP. Although we present our model from a generative perspective, we show in the description of the framework that discriminative techniques can be integrated. Inference in our model is described in section "Inference" . There, we present a chart-driven agenda parser that can leverage the semantic prior to guide its search. Finally, in section "Results" , we evaluate our parser on two relation-extraction tasks: the first is a task to extract simple predicate-argument representations from SVO sentences, and the second is a temporally-scoped relation extraction task that demonstrates our parser's ability to model the morphology of individual words, leading to improved generalization performance over words. Moreover, we demonstrate that the inclusion of background knowledge from a knowledge base improves parsing performance on these tasks. The key contributions of this article are:
Background
Our model is an extension of context-free grammars (CFGs) BIBREF18 that couples syntax and semantics. To generate a sentence in our framework, the semantic statement is first drawn from a prior. A grammar then recursively constructs a syntax tree top-down, randomly selecting production rules from distributions that depend on the semantic statement. We present a particular incarnation of a grammar in this framework, where hierarchical Dirichlet processes (HDPs) BIBREF19 are used to select production rules randomly. The application of HDPs in our setting is novel, requiring a new inference technique.
The use of the term “generative” does not refer to the Chomskian tradition of generative grammar BIBREF20 , although our approach does fall broadly within that framework. Rather, it refers to the fact that our model posits a probabilistic mechanism by which sentences are generated (by the speaker). Performing probabilistic inference under this model yields a parsing algorithm (the listener). This generative approach to modeling grammar underscores the duality between language generation and language understanding.
Our grammar can be related to synchronous CFGs (SCFGs) BIBREF21 , which have been extended to perform semantic parsing BIBREF22 , BIBREF23 , BIBREF24 . However, in established use, SCFGs describe the generation of the syntactic and semantic components of sentences simultaneously, which makes the assumption that the induced probability distributions of the semantic and syntactic components factorize in a “parallel” manner. Our model instead describes the generation of the semantic component as a step with occurs prior to the syntactic component. This can be captured in SCFGs as a prior on the semantic start symbol, making no factorization assumptions on this prior. This is particularly useful when employing richer prior distributions on the semantics, such as a model of context or a knowledge base.
Adaptor grammars BIBREF25 provide a framework that can jointly model the syntactic structure of sentences in addition to the morphologies of individual words BIBREF26 . Unlike previous work with adaptor grammars, our method couples syntax with semantics, and can be guided by background knowledge via the semantic prior. We will demonstrate how this can be leveraged in our framework to model the morphology of individual verbs in a temporally-scoped relation extraction task. Cohen10 show how to perform dependency grammar induction using adaptor grammars. While grammar induction in our framework constitutes an interesting research problem, we do not address it in this work.
As in other parsing approaches, an equivalence can be drawn between our parsing problem and the problem of finding shortest paths in hypergraphs BIBREF27 , BIBREF28 , BIBREF29 , BIBREF30 , BIBREF31 . Our algorithm can then be understood as an application of $\textrm {A}^*$ search for the $k$ -best paths in a very large hypergraph.
Our parser incorporates prior knowledge to guide its search, such as from an ontology and the set of beliefs in a knowledge base. Using this kind of approach, the parser can be biased to find context-appropriate interpretations in otherwise ambiguous or terse utterances. While systems such as DurrettK14, NakasholeM15, KimMoldovan1995, and Salloum09 use background knowledge about the semantic types of different noun phrases to improve their ability to perform entity linking, co-reference resolution, prepositional phrase attachment, information extraction, and question answering, and systems such as RatinovR12, DurrettK14, and ProkofyevTLVDC15 link noun phrases to Wikipedia entries to improve their ability to resolve co-references, these uses of background knowledge remain fragmentary. Krishnamurthy2014 developed a CCG parser that incorporates background knowledge from a knowledge base during training through distant supervision, but their method is not able to do so during parsing. Our parser can be trained once, and then applied to a variety of settings, each with a different context or semantic prior.
Hierarchical Dirichlet processes
A core component of our statistical model is the Dirichlet process (DP) BIBREF32 , which can be understood as a distribution over probability distributions. If a distribution $G$ is drawn from a DP, we can write $G\sim \text{DP}(\alpha ,H)$ , where the DP is characterized by two parameters: a concentration parameter $\alpha >0$ and a base distribution $H$ . The DP has the useful property that $\mathbb {E}[G] = H$ , and the concentration parameter $\alpha $ describes the “closeness” of $G$ to the base distribution $H$ . In typical use, a number of parameters $\theta _i$ are drawn from a discrete distribution $G$ , which is itself drawn from a Dirichlet process. The observations $G\sim \text{DP}(\alpha ,H)$0 are drawn using the parameters $G\sim \text{DP}(\alpha ,H)$1 from another distribution $G\sim \text{DP}(\alpha ,H)$2 . This may be written as:
$$G &\sim \text{DP}(\alpha , H), \\ \theta _1,\dots ,\theta _n &\sim G, \\ y_i &\sim F(\theta _i),$$   (Eq. 6)
for $i=1,\hdots ,n$ . In our application, we will define $H$ to be a finite Dirichlet distribution and $F$ is a categorical distribution. $G$ can be marginalized out in the model above, resulting in the Chinese restaurant process representation BIBREF33 :
$$\phi _1, \phi _2, \dots &\sim H, \\ z_i &= {\left\lbrace \begin{array}{ll} j & \text{with probability } \frac{\#\lbrace k < i : z_k = j\rbrace }{\alpha + i - 1}, \\ j^{new} & \text{with probability } \frac{\alpha }{\alpha + i - 1}, \end{array}\right.} \\ \theta _i &= \phi _{z_i} \text{ for } i = 1, \dots , n, \\ y_i &\sim F(\theta _i),$$   (Eq. 7)
where $z_1 = 1$ , $j^{new} = \max \lbrace z_1,\dots ,z_{i-1}\rbrace  + 1$ is the indicator of a new table, and the quantity $\#\lbrace k < i : z_k = j\rbrace $ is the number of observations that were assigned to table $j$ . The analogy is to imagine a restaurant where customers enter one at a time. Each customer chooses to sit at table $j$ with probability proportional to the number of people currently sitting at table $j$ , or at a new table $j^{new}$ with probability proportional to $\alpha $ . The $i^{th}$ customer's choice is represented as $z_i$ . As shown in later sections, this representation of the DP is amenable to inference using Markov chain Monte Carlo (MCMC) methods BIBREF34 , BIBREF35 .
The hierarchical Dirichlet process (HDP) is an extension of the Dirichlet process for use in hierarchical modeling BIBREF19 . An advantage of this approach is that statistical strength can be shared across nodes that belong to the same subtree. In an HDP, every node $\textbf {n}$ in a fixed tree $T$ is associated with a distribution $G^\textbf {n}$ , and:
$$G^\textbf {0} &\sim \text{DP}(\alpha ^{\textbf {0}}, H), \\ G^\textbf {n} &\sim \text{DP}(\alpha ^{\textbf {n}}, G^{\pi (\textbf {n})}),$$   (Eq. 8)
where $\pi (\textbf {n})$ is the parent node of $\textbf {n}$ , and $\textbf {0}$ is the root of $T$ . In our application, the base distribution at the root $H$ is Dirichlet. We can draw observations $y_1,\hdots ,y_n$ from the HDP, given a sequence $x_1,\hdots ,x_n$ of $n$ paths from the root $\textbf {0}$ to a leaf:
$$\theta _i &\sim G^{x_i}, \\ y_i &\sim F(\theta _i),$$   (Eq. 9)
for $i=1,\hdots ,n$ . For notational brevity, we write this equivalently as $y_i\sim \text{HDP}(x_i,T)$ .
Just as marginalizing the Dirichlet process yields the Chinese restaurant process, marginalizing the HDP yields the Chinese restaurant franchise (CRF). For every node in the HDP tree $\textbf {n} \in T$ , there is a “Chinese restaurant” consisting of an infinite number of tables. Every table $i$ in this restaurant at node $\textbf {n}$ is assigned to a table in the parent restaurant. The assignment variable $z_i^\textbf {n}$ is the index of the parent table to which table $i$ in node $\textbf {n}$ is assigned.
$$\phi _1^\textbf {0}, \phi _2^\textbf {0}, \dots &\sim H, \\ \text{ for every node } \textbf {n} \in T, \hspace{14.22636pt} z_i^\textbf {n} &= {\left\lbrace \begin{array}{ll} j & \text{with probability } \propto n^{\pi (\textbf {n})}_j, \\ j^{new} & \text{with probability } \propto \alpha ^{\pi (\textbf {n})}, \end{array}\right.} \\ \phi _i^\textbf {n} &= \phi _{z_i^\textbf {n}}^{\pi (\textbf {n})},$$   (Eq. 10)
where $\pi (\textbf {n})$ is the parent of node $\textbf {n}$ , and $n^{\pi (\textbf {n})}_j$ is the current number of customers at node $\pi (\textbf {n})$ sitting at table $j$ . We are mildly abusing notation here, since $n^{\pi (\textbf {n})}_j$ and $n^{\pi (\textbf {n})}$ refer to the number of customers at the time $z_i^\textbf {n}$ is drawn (which increases as additional $z_i^\textbf {n}$ are drawn). To draw the observation $y_i$ , we start with the leaf node at the end of the path $\textbf {n}$0 :
$$\theta _i &= \phi ^{x_i}_k, \\ y_i &\sim F(\theta _i),$$   (Eq. 11)
where $k - 1 = \#\lbrace j < i : x_j = x_i\rbrace $ is the number of previous observations drawn from node $x_i$ .
Inference
In this section, we describe our method for performing posterior inference in the HDP. Let $\mathbf {z} = \lbrace z^\textbf {n}_i : \textbf {n} \in T, i = 1,2,\hdots \rbrace $ be the set of table assignment variables in the HDP. If the distributions $H$ and $F$ are conditionally conjugate, as they are in our application, the $\mathbf {\phi }$ variables can be integrated out in closed form:
$$p(\mathbf {z}|\mathbf {x},\mathbf {y}) = p(\mathbf {x}) p(\mathbf {z}) \int p(\mathbf {y}|\mathbf {x},\mathbf {z},\mathbf {\phi }) d\mathbf {\phi }.$$   (Eq. 13)
The posterior $p(\mathbf {z}|\mathbf {x},\mathbf {y})$ is intractable to compute exactly, and so we approximate it by sampling. We obtain samples from $\mathbf {z}|\mathbf {x},\mathbf {y}$ by performing collapsed Gibbs sampling as described in section 5.1 of journals/jasa/Teh06: we repeatedly sample $\mathbf {z}$ from its conditional distribution, with $\mathbf {\phi }$ integrated out:
$$z^\textbf {n}_i | \mathbf {x}, \mathbf {y}, z^\textbf {n}_{-i} = {\left\lbrace \begin{array}{ll} j &\text{with prob.} \propto \#\lbrace k\ne i : z^\textbf {n}_k = j\rbrace  \cdot p(y^\textbf {n}_i | \mathbf {x}, y^\textbf {n}_{-i}, z^\textbf {n}_{-i}, z^\textbf {n}_i = j), \\ j^{new} &\text{with prob.} \propto \alpha ^\textbf {n} \cdot p(y^\textbf {n}_i | \mathbf {x}, y^\textbf {n}_{-i}, z^\textbf {n}_{-i}, z^\textbf {n}_i = j^{new}), \end{array}\right.} $$   (Eq. 14)
where $y^\textbf {n}_i$ is the set of “descendant” observations of table $i$ in node $\textbf {n}$ (this includes observations assigned directly to the table, in addition to those assigned to tables further down in the hierarchy which themselves are assigned to this table), $y^\textbf {n}_{-i} = \mathbf {y} \setminus y^\textbf {n}_i$ is the set of all other observations, and $z^\textbf {n}_{-i} = \mathbf {z} \setminus z^\textbf {n}_i$ is the set of all other table assignment variables. Computing $p(y^\textbf {n}_i | \mathbf {x}, y^\textbf {n}_{-i}, z^\textbf {n}_{-i}, z^\textbf {n}_i = j)$ is straightforward since we can follow the chain of table assignments to the root. Let $r^\textbf {n}_i$ be the root cluster assignment of the table $i$ at node $\textbf {n}$ . In fact, we found it advantageous for performance to keep track of the root cluster assignments $\mathbf {r}$ for every table in the hierarchy. Thus, when $i$0 , it must be the case that $i$1 were drawn from $i$2 with parameter $i$3 .
Computing $p(y^\textbf {n}_i | \mathbf {x}, y^\textbf {n}_{-i}, z^\textbf {n}_{-i}, z^\textbf {n}_i = j^{new})$ requires marginalizing over the assignment of the new table $z^{\pi (\textbf {n})}_{j^{new}}$ :
$$p(y^\textbf {n}_i | \mathbf {x}, y^\textbf {n}_{-i}, z^\textbf {n}_{-i}, z^\textbf {n}_i = j^{new}) = &\sum _{k=1}^{m^{\pi (\textbf {n})}} \frac{n_k^{\pi (\textbf {n})}}{n^{\pi (\textbf {n})} + \alpha ^{\pi (\textbf {n})}} p(y^\textbf {n}_i | \mathbf {x}, y^\textbf {n}_{-i}, z^\textbf {n}_{-i}, z^{\pi (\textbf {n})}_{j^{new}} = k) \nonumber \\ &+ \frac{\alpha ^{\pi (\textbf {n})}}{n^{\pi (\textbf {n})} + \alpha ^{\pi (\textbf {n})}} p(y^\textbf {n}_i | \mathbf {x}, y^\textbf {n}_{-i}, z^\textbf {n}_{-i}, z^{\pi (\textbf {n})}_{j^{new}} = k^{new}),$$   (Eq. 15)
where $m^{\pi (\textbf {n})}$ is the number of occupied tables at the node $\pi (\textbf {n})$ . At the root node $\pi (\textbf {n}) = \textbf {0}$ , the above probability is just the prior of $y^\textbf {n}_i$ . We observe that the above probabilities are linear functions of the likelihoods $p(y^\textbf {n}_i | \mathbf {x}, y^\textbf {n}_{-i}, z^\textbf {n}_{-i}, r^\textbf {n}_i = k)$ for various root cluster assignments $r^\textbf {n}_i = k$ . Implemented naively, generating a single sample from equation 14 can take time linear in the number of clusters at the root, which would result in a quadratic-time algorithm for a single Gibbs iteration over all $\mathbf {z}$ . However, we can exploit sparsity in the root cluster assignment likelihoods to improve performance. When $H = \text{Dir}(\beta )$ is a Dirichlet distribution and $F$ is a categorical, then the collapsed root cluster assignment likelihood is:
$$p(y^\textbf {n}_i | \mathbf {x}, y^\textbf {n}_{-i}, z^\textbf {n}_{-i}, r^\textbf {n}_i = k) = \frac{\prod _t \left( \beta _t + \#\lbrace t \in y^\textbf {0}_k\rbrace  \right)^{(\#\lbrace t \in y^\textbf {n}_i\rbrace )}}{\left(\sum _t \beta _t + \# y^\textbf {0}_k \right)^{(\# y^\textbf {n}_i)}}.$$   (Eq. 16)
Here, $a^{(b)}$ is the rising factorial $a(a + 1)(a + 2)\hdots (a + b - 1) = \frac{\Gamma (a + b)}{\Gamma (a)}$ , and $\#\lbrace t \in y^\textbf {n}_i\rbrace $ is the number of elements in $y^\textbf {n}_i$ with value $t$ . Notice that the denominator depends only on the sizes and not on the contents of $y^\textbf {n}_i$ and $y^\textbf {0}_k$ . Caching the denominator values for common sizes of $y^\textbf {n}_i$ and $y^\textbf {0}_k$ can allow the sampler to avoid needless recomputation. This is especially useful in our application since many of the tables at the root tend to be small. Similarly, observe that the numerator factor is 1 for values of $t$ where $a(a + 1)(a + 2)\hdots (a + b - 1) = \frac{\Gamma (a + b)}{\Gamma (a)}$0 . Thus, the time required to compute the above probability is linear in the number of unique elements of $a(a + 1)(a + 2)\hdots (a + b - 1) = \frac{\Gamma (a + b)}{\Gamma (a)}$1 , which can improve the scalability of our sampler. We perform the above computations in log space to avoid numerical overflow.
In previous uses of the HDP, the paths $x_i$ are assumed to be fixed. For instance, in document modeling, the paths correspond to documents or predefined categories of documents. In our application, however, the paths may be random. In fact, we will later show that our parser heavily relies on the posterior predictive distribution over paths, where the paths correspond to semantic parses. More precisely, given a collection of training observations $\mathbf {y} = \lbrace y_1,\hdots ,y_n\rbrace $ with their paths $\mathbf {x} = \lbrace x_1,\hdots ,x_n\rbrace $ , we want to compute the probability of a new path $x^{new}$ given a new observation $y^{new}$ :
$$p(x^{new}|y^{new},\mathbf {x},\mathbf {y}) &\propto p(x^{new}) \int p(y^{new}|\mathbf {z},x^{new}) p(\mathbf {z}|\mathbf {x},\mathbf {y}) d\mathbf {z},  \\ &\approx \frac{p(x^{new})}{N_{samples}} \sum _{\mathbf {z}^* \sim \mathbf {z}|\mathbf {x},\mathbf {y}} p(y^{new}|\mathbf {z}^*,x^{new}). $$   (Eq. 18)
Once we have the posterior samples $\mathbf {z}^*$ , we can compute the quantity $p(y^{new}|\mathbf {z}^*,x^{new})$ by marginalizing over the table assignment for the new observation $y$ :
$$p(y^{new}|\mathbf {z}^*,x^{new}) = &\sum _{j=1}^{m^{x^{new}}} \frac{n_j^{x^{new}}}{n^{x^{new}} + \alpha ^{x^{new}}} \hspace{2.84544pt} p(y^{new} | \mathbf {z}^*, \theta ^{new} = \phi ^{x^{new}}_j) \nonumber \\ &+ \frac{\alpha ^{x^{new}}}{n^{x^{new}} + \alpha ^{x^{new}}} \hspace{2.84544pt} p(y^{new} | \mathbf {z}^*, \theta ^{new} = \phi ^{x^{new}}_{j^{new}}).$$   (Eq. 19)
Here, $m^{x^{new}}$ is the number of occupied tables at node $x^{new}$ , $n^{x^{new}}_j$ is the number of customers sitting at table $j$ at node $x^{new}$ , and $n^{x^{new}}$ is the total number of customers at node $x^{new}$ . The first term $p(y^{new} | \mathbf {z}^*, \theta ^{new} = \phi ^{x^{new}}_j)$ can be computed since the $j^{th}$ table exists and is assigned to a table in its parent node, which in turn is assigned to a table in its parent node, and so on. We can follow the chain of table assignments to the root. In the second term, the observation is assigned to a new table, whose assignment is unknown, and so we marginalize again over the assignment in the parent node for this new table:
$$p(y^{new} | \mathbf {z}^*, \theta ^{new} = \phi ^{x^{new}}_{j^{new}}) = &\sum _{j=1}^{m^{\pi (x^{new})}} \frac{n_j^{\pi (x^{new})}}{n^{\pi (x^{new})} + \alpha ^{\pi (x^{new})}} \hspace{5.69046pt} p\left(y^{new} \Big | \mathbf {z}^*, \theta ^{new} = \phi _j^{\pi (x^{new})}\right) \nonumber \\ &+ \frac{\alpha ^{\pi (x^{new})}}{n^{\pi (x^{new})} + \alpha ^{\pi (x^{new})}} \hspace{5.69046pt} p\left(y^{new} \Big | \mathbf {z}^*, \theta ^{new} = \phi _{j^{new}}^{\pi (x^{new})}\right),$$   (Eq. 20)
where $\pi (x^{new})$ is the parent node of $x^{new}$ . Again, the probability in the first term can be computed as before, but the probability in the second term depends on the assignment of the new table, which is unknown. Thus, since it is possible that a new table will be created at every level in the hierarchy up to the root, we can apply this formula recursively. At the root $\textbf {0}$ , the probability $p(y^{new} | \mathbf {z}^*, \theta ^{new} = \phi _{j^{new}}^\textbf {0})$ is just the prior probability of $y^{new}$ .
If the tree $T$ is small, it is straightforward to compute the quantity in equation for every path $x^{new}$ in the tree, using the method described above. In our application however, the size of $T$ depends on the size of the ontology, and may easily become very large. In this case, the naïve approach becomes computationally infeasible. As such, we develop an algorithm to incrementally find the $k$ best paths that maximize the quantity in equation . For sparse distributions, where most of the probability mass is concentrated in a small number of paths $x^{new}$ , this algorithm can effectively characterize the predictive distribution in equation 18 . The algorithm is essentially a search over nodes in the tree, starting at the root and descending the nodes of the tree $T$ , guided through paths of high probability. Each search state $\texttt {s}$ consists of the following fields:
$\texttt {s.n}$ is the current position of the search in the tree.
$\texttt {s.v}$ is an array of probability scores of length $N_{samples}$ . Each element in this array represents the probability of drawing the observation $y^{new}$ from the current node $\texttt {s.n}$ , and thus is identical to the probability of assigning $y^{new}$ to a new table at any child node of $\texttt {s.n}$ . This is useful to compute the quantity in equation using the recursive method as described above.
The search is outlined in algorithm UID17 . We observe that the quantity in equation is a sum of independent functions, each being a linear combination of the terms $p(y^{new}|\mathbf {z}^*_i,\theta ^{new} = \phi _j^\textbf {n})$ over the tables available at node $\textbf {n}$ and the new table $p(y^{new}|\mathbf {z}^*_i,\theta ^{new} = \phi _{j^{new}}^\textbf {n})$ (this latter probability is stored in $\texttt {s.v}_i$ ). Thus, the upper bound on equation over all paths that pass through node $\texttt {s.n}$ is:
$$\max _{\lbrace x^{new}:\texttt {s.n} \in x^{new}\rbrace } \frac{p(x^{new})}{N_{samples}} \sum _{i=1}^{N_{samples}} \max _{j=1,\hdots ,m^\texttt {s.n}} \left\lbrace  p(y^{new}|\mathbf {z}^*_i,\theta ^{new}=\phi _j^\texttt {s.n}) , \texttt {s.v}_i \right\rbrace . $$   (Eq. 23)
We sort elements in the priority queue using this expression.
Fnfunction IfElseIfElseif elifelse Whilewhile{ Repeatrepeatuntil
initialize priority queue with initial state $\texttt {s}$ $\texttt {s.n} \leftarrow \textbf {0}$ *[h]start at the root $\normalfont i=1,\hdots ,N_{samples},$ $\texttt {s.v}_i \leftarrow \sum _{j=1}^{m^\textbf {0}} \frac{n_j^\textbf {0}}{n^\textbf {0} + \alpha ^\textbf {0}} p(y^{new}|\mathbf {z}^*_i, \theta ^{new} = \phi _j^\textbf {0}) + \frac{\alpha ^\textbf {0}}{n^\textbf {0} + \alpha ^\textbf {0}} p(y^{new} | \mathbf {z}^*_i, \theta ^{new} = \phi ^\textbf {0}_{j^{new}})$ there are $k$ completed paths pop state s from the priority queue $\normalfont \texttt {s.n}$ is a leaf complete the path s.n with probability $\frac{p\lbrace x^{new} = \texttt {s.n}\rbrace }{N_{samples}} \sum _{i=1}^{N_{samples}} \texttt {s.v}_i$ child node $\normalfont \textbf {c}$ of $\normalfont \texttt {s.n}$ , create new search state $\texttt {s}^*$ $\texttt {s.n} \leftarrow \textbf {0}$0 $\texttt {s.n} \leftarrow \textbf {0}$1 $\texttt {s.n} \leftarrow \textbf {0}$2 push $\texttt {s.n} \leftarrow \textbf {0}$3 onto priority queue with key in equation 23 Search algorithm to find the $\texttt {s.n} \leftarrow \textbf {0}$4 best paths in the HDP that maximize the quantity in equation . As a result, once the algorithm has completed $\texttt {s.n} \leftarrow \textbf {0}$5 items, we are guaranteed that the search has found $\texttt {s.n} \leftarrow \textbf {0}$6 best paths. Thus, an “iterator” data structure can be efficiently implemented using this algorithm, which returns paths $\texttt {s.n} \leftarrow \textbf {0}$7 in order of decreasing predictive probability, with the first item being optimal. The search algorithm can be modified for other representations of the HDP, and can be extended to the case where $\texttt {s.n} \leftarrow \textbf {0}$8 and $\texttt {s.n} \leftarrow \textbf {0}$9 are not conjugate. It may also be incorporated into a larger inference procedure to jointly infer the paths $\normalfont i=1,\hdots ,N_{samples},$0 and the latent variables in the HDP. It is also straightforward to compute predictive probabilities where the path $\normalfont i=1,\hdots ,N_{samples},$1 is restricted to a subset of paths $\normalfont i=1,\hdots ,N_{samples},$2 : $\normalfont i=1,\hdots ,N_{samples},$3 . To do so, the algorithm is restricted to only expand nodes that belong to paths in $\normalfont i=1,\hdots ,N_{samples},$4 .
An important concern when performing inference with very large trees $T$ is that it is not feasible to explicitly store every node in memory. Fortunately, collapsed Gibbs sampling does not require storing nodes whose descendants have zero observations. In addition, algorithm UID17 can be augmented to avoid storing these nodes, as well. To do so, we make the observation that for any node $\textbf {n} \in T$ in the tree whose descendants have no observations, $\textbf {n}$ will have zero occupied tables. Therefore, the probability $p(y^{new}|\mathbf {z}^*,x^{new}) = p(y^{new}|\mathbf {z}^*,\theta ^{new}=\phi ^\textbf {n}_{j^{new}})$ is identical for any path $x^{new}$ that passes through $\textbf {n}$ . Thus, when the search reaches node $\textbf {n}$ , it can simultaneously complete all paths $x^{new}$ that pass through $\textbf {n}$ , and avoid expanding nodes with zero observations among its descendants. As a result, we only need to explicitly store a number of nodes linear in the size of the training data, which enables practical inference with very large hierarchies.
There is a caveat that arises when we wish to compute a joint predictive probability $p(x^{new}_1, \hdots , x^{new}_k | y^{new}_1, \hdots , y^{new}_k, \mathbf {x}, \mathbf {y})$ , where we have multiple novel observations. Re-writing equation 18 in this setting, we have:
$$p(x^{new}_1, \hdots , x^{new}_k &| y^{new}_1, \hdots , y^{new}_k, \mathbf {x}, \mathbf {y}) \nonumber \\ &\propto p(\mathbf {x}^{new}) \int p(y^{new}_1,\hdots ,y^{new}_k|\mathbf {z}^*,\mathbf {x}^{new}) p(\mathbf {z}|\mathbf {x},\mathbf {y})d\mathbf {z}. $$   (Eq. 24)
For the CRF, the joint likelihood $p(y^{new}_1,\hdots ,y^{new}_k|\mathbf {z}^*,\mathbf {x}^{new})$ does not factorize, since the observations are not independent (they are exchangeable). One workaround is to use a representation of the HDP where the joint likelihood factorizes, such as the direct assignment representation BIBREF19 . Another approach is to approximate the joint likelihood with the factorized likelihood. In our parser, we instead make the following approximation:
$$p(y^{new}_1,\hdots ,y^{new}_k | \mathbf {x}^{new},\mathbf {x},\mathbf {y}) &= \prod _{i=1}^k p(y^{new}_i | y^{new}_1,\hdots ,y^{new}_{i-1}, \mathbf {x}^{new},\mathbf {x},\mathbf {y}) \\ &\approx \prod _{i=1}^k p(y^{new}_i | \mathbf {x}^{new},\mathbf {x},\mathbf {y}). $$   (Eq. 25)
Substituting into equation 24 , we obtain:
$$p(\mathbf {x}^{new} | \mathbf {y}^{new}, \mathbf {x}, \mathbf {y}) \propto p(\mathbf {x}^{new}) \prod _{i=1}^k \int p(y^{new}_i|\mathbf {z}^*,\mathbf {x}^{new}) p(\mathbf {z}|\mathbf {x},\mathbf {y})d\mathbf {z}.$$   (Eq. 26)
When the size of the training data $(\mathbf {x},\mathbf {y})$ is large with respect to the test data $(\mathbf {x}^{new},\mathbf {y}^{new})$ , the approximation works well, which we also find to be the case in our experiments.
Generative semantic grammar
We present a generative model of text sentences. In this model, semantic statements are generated probabilistically from some higher-order process. Given each semantic statement, a formal grammar selects text phrases, which are concatenated to form the output sentence. We present the model such that it remains flexible with regard to the semantic formalism. Even though our grammar can be viewed as an extension of context-free grammars, it is important to note that our model of grammar is only conditionally context-free, given the semantic statement. Otherwise, if the semantic information is marginalized out, the grammar is sensitive to context.
Definition
Let $\mathcal {N}$ be a set of nonterminals, and let $\mathcal {W}$ be a set of terminals. Let $\mathbf {R}$ be a set of production rules which can be written in the form $\textrm {A} \rightarrow \textrm {B}_1 \hdots \textrm {B}_k$ where $\textrm {A}\in \mathcal {N}$ and $\textrm {B}_1,\hdots ,\textrm {B}_k\in \mathcal {W}{2mu}\cup {2mu}\mathcal {N}$ . The tuple $(\mathcal {W},\mathcal {N},\mathbf {R})$ is a context-free grammar (CFG) BIBREF18 .
We couple syntax with semantics by augmenting the production rules $\mathbf {R}$ . In every production rule $\textrm {A} \rightarrow \textrm {B}_1 \hdots \textrm {B}_k$ in $\mathbf {R}$ , we assign to every right-hand side symbol $B_i$ a surjective operation $f_i : \mathcal {X}_A\mapsto \mathcal {X}_{B_i}$ that transforms semantic statements, where $\mathcal {X}_A$ is the set of semantic statements associated with the symbol $\textrm {A}$ and $\mathcal {X}_{B_i}$ is the set of semantic statements associated with the symbol $\textrm {B}_i$ . Intuitively, the operation describes how the semantic statement is “passed on” to the child nonterminals in the generative process. During parsing, these operations will describe how simpler semantic statements combine to form larger statements, enabling semantic compositionality. For example, suppose we have a semantic statement $x = \textit {has\_color(reptile:frog,color:green)}$ and the production rule $\textrm {A} \rightarrow \textrm {B}_1 \hdots \textrm {B}_k$0 . We can pair the semantic operation $\textrm {A} \rightarrow \textrm {B}_1 \hdots \textrm {B}_k$1 with the $\textrm {A} \rightarrow \textrm {B}_1 \hdots \textrm {B}_k$2 in the right-hand side such that $\textrm {A} \rightarrow \textrm {B}_1 \hdots \textrm {B}_k$3 selects the subject argument. Similarly, we can pair the semantic operation $\textrm {A} \rightarrow \textrm {B}_1 \hdots \textrm {B}_k$4 with the $\textrm {A} \rightarrow \textrm {B}_1 \hdots \textrm {B}_k$5 in the right-hand side such that $\textrm {A} \rightarrow \textrm {B}_1 \hdots \textrm {B}_k$6 is the identity operation. The augmented production rule is $\textrm {A} \rightarrow \textrm {B}_1 \hdots \textrm {B}_k$7 and the set of augmented rules is $\textrm {A} \rightarrow \textrm {B}_1 \hdots \textrm {B}_k$8 . In parsing, we require the computation of the inverse of semantic operations, which is the preimage of a given semantic statement $\textrm {A} \rightarrow \textrm {B}_1 \hdots \textrm {B}_k$9 . Continuing the example above, $\mathbf {R}$0 returns a set that contains the statement $\mathbf {R}$1 in addition to statements like $\mathbf {R}$2 .
To complete the definition of our grammar, we need to specify the method that, given a nonterminal $\mathrm {A} \in \mathcal {N}$ and a semantic statement $x \in \mathcal {X}_A$ , selects a production rule from the set of rules in $\mathbf {R}^*$ with the left-hand side nonterminal $A$ . To accomplish this, we define $\texttt {select}_{A,x}$ as a distribution over rules from $\mathbf {R}^*$ that has $\textrm {A}$ as its left-hand side, dependent on $x$ . We will later provide a number of example definitions of this $\texttt {select}_{A,x}$ distribution. Thus, a grammar in our framework is fully specified by the tuple $(\mathcal {W},\mathcal {N},\mathbf {R}^*,\texttt {select})$ .
Note that other semantic grammar formalisms can be fit into this framework. For example, in categorical grammars, a lexicon describes the mapping from elementary components of language (such as words) to a syntactic category and a semantic meaning. Rules of inference are available to combine these lexical items into (tree-structured) derivations, eventually resulting in a syntactic and semantic interpretation of the full sentence BIBREF36 , BIBREF37 . In our framework, we imagine this process in reverse. The set $\mathcal {X}_S$ is the set of all derivable semantic statements with syntactic category $S$ . The generative process begins by selecting one statement from this set $x \in \mathcal {X}_S$ . Next, we consider all applications of the rules of inference that would yield $x$ , with each unique application of an inference rule being equivalent to a production rule in our framework. We select one of these production rules according to our generative process and continue recursively. The items in the lexicon are equivalent to preterminal production rules in our framework. Thus, the generative process below describes a way to endow parses in categorical grammar with a probability measure. This can be used, for example, to extend earlier work on generative models with CCG BIBREF38 , BIBREF39 . Different choices of the $\texttt {select}$ distribution induce different probability distributions over parses.
We do not see a straightforward way to fit linear or log-linear models over full parses into our framework, where a vector of features can be computed for each full parse BIBREF40 , BIBREF41 . This is due to our assumption that, given the semantic statement, the probability of a parse factorizes over the production rules used to construct that parse. However, the select distribution can be defined using linear and log-linear models, as we will describe in section "Selecting production rules" .
Generative process
The process for generating sentences in this framework begins by drawing a semantic statement $x\in \mathcal {X}_S$ where $\textrm {S}$ is the root nonterminal. Thus, there is a prior distribution $p(x)$ for all $x\in \mathcal {X}_S$ . Next, the syntax is generated top-down starting at $\textrm {S}$ . We draw a production rule with $\textrm {S}$ as the left-hand side from $\texttt {select}_{S,x}$ . The semantic transformation operations $f_i$ are applied to $x$ and the process is repeated for the right-hand side nonterminals. More concretely, we define the following operation $\texttt {expand}$ which takes two arguments: a symbol $\textrm {S}$0 and a semantic statement $\textrm {S}$1 .
FExpandexpand FYieldyield FSelectselect tworuled
[H] $x$ , A ${\rm A}\in \mathcal {W}$ *[h]simply return the word if A is a terminal
A *[h]select a production rule with form $\textrm {A}\rightarrow \textrm {B}_1,\hdots ,\textrm {B}_k$
$(\textrm {A}, \textrm {B}_1, \hdots , \textrm {B}_k, f_1, \hdots , f_k) \sim _{\textrm {A},x}$ $((f_1(x), \textrm {B}_1), \hdots , (f_k(x), \textrm {B}_k))$
The yield operation concatenates strings into a single output string. Then, the output sentence $y$ is generated simply by $y=\texttt {expand}(x, \textrm {S})$ . Depending on the application, we may require that the generative process capitalizes the first letter of the output sentence, and/or appends terminating punctuation to the end. A noise model may also be appended to the generative process. The above algorithm may be easily extended to also return the full syntax tree.
Selecting production rules
There are many possible choices for the $\texttt {select}$ distribution. The most straightforward is to define a categorical distribution over the available production rules, and simply draw the selected rule from this distribution. The result would be a simple extension of probabilistic context-free grammars (PCFGs) that couples semantics with syntax. However, this would remove any dependence between the semantic statement and the production rule selection.
To illustrate the importance of this dependence, consider generating a sentence with the semantic statement athlete_plays_sport(athlete:roger_federer,sport:tennis) using the grammar in figure 2 (the process is graphically depicted in figure 3 ). We start with the root nonterminal $\textrm {S}$ :
We can only select the first production rule, and so we apply the semantic operation select_arg1 on the semantic statement to obtain athlete:roger_federer for the right-hand side nonterminal $\textrm {N}$ . We apply the semantic operation delete_arg1 to obtain athlete_plays_sport( $\cdot $ ,sport:tennis) for $\textrm {VP}$ .
Expanding $\textrm {N}$ , we select a terminal symbol given the semantic statement athlete:roger_federer. Suppose “Andre Agassi” is returned.
Now, we expand the $\textrm {VP}$ symbol. We draw from $\texttt {select}_{\textrm {VP}}$ to choose one of the two available production rules. Suppose the rule $\textrm {VP} \rightarrow \textrm {V \hspace{2.84544pt} N}$ is selected. Thus, we apply the identity operation for the $\textrm {V}$ nonterminal to obtain athlete_plays_sport( $\cdot $ ,sport:tennis). We similarly apply select_arg2 for the $\textrm {N}$ nonterminal to obtain sport:tennis.
We expand the $\textrm {V}$ nonterminal, drawing from $\texttt {select}_{\textrm {V}}$ on the semantic statement athlete_plays_sport( $\cdot $ ,sport:tennis). Suppose “plays” is returned.
Finally, we expand the $\textrm {N}$ nonterminal, drawing from $\texttt {select}_{\textrm {N}}$ with the statement sport:tennis. Suppose “tennis” is returned. We concatenate all returned strings to form the sentence “Andre Agassi plays tennis.”
However, now consider generating another sentence with the same grammar for the statement athlete_plays_sport(athlete:roger_federer, sport:swimming). In UID32 of the above process, the select distribution would necessarily have to depend on the semantic statement. In English, the probability of observing a sentence of the form $\textrm {N} \hspace{2.84544pt} \textrm {V} \hspace{2.84544pt} \textrm {N}$ ('Rachmaninoff makes music') versus $\textrm {N} \hspace{2.84544pt} \textrm {V}$ ('Rachmaninoff composes') depends on the underlying semantic statement.
To capture this dependence, we use HDPs to define the select distribution. Every nonterminal $\textrm {A}\in \mathcal {N}$ is associated with an HDP, and in order to fully specify the grammar, we need to specify the structure of each HDP tree. Let $T_A$ be the tree associated with the nonterminal $\textrm {A}$ . The model is flexible with how the trees are defined, but we construct trees with the following method. First, select $m$ discrete features $g_1,\hdots ,g_m$ where each $g_i : \mathcal {X} \mapsto \mathbb {Z}$ and $\mathbb {Z}$ is the set of integers. These features operate on semantic statements. For example, suppose we restrict the space of semantic statements to be the set of single predicate instances (triples). The relations in an ontology can be assigned unique integer indices, and so we may define a semantic feature as a function which simply returns the index of the predicate given a semantic statement. We construct the HDP tree $T_A$ starting with the root, we add a child node for every possible output of $g_1$ . We repeat the process recursively, constructing a complete tree of depth $m + 1$ .
As an example, we will construct a tree for the nonterminal $\textrm {VP}$ for the example grammar in figure 2 . Suppose in our ontology, we have the predicates athlete_plays_sport and musician_plays_instrument, labeled 0 and 1, respectively. The ontology also contains the concepts athlete:roger_federer, sport:tennis, and sport:swimming, also labeled 0, 1, and 2, respectively. We define the first feature $g_1$ to return the predicate index. The second feature $g_2$ returns the index of the concept in the second argument of the semantic statement. The tree is constructed starting with the root, we add a child node for each predicate in the ontology: athlete_plays_sport and musician_plays_instrument. Next, for each child node, we add a grandchild node for every concept in the ontology: athlete:roger_federer, sport:tennis, and sport:swimming. The resulting tree $T_{VP}$ has depth 2, with a root node with 2 child nodes, and each child node has 3 grandchild nodes. This construction enables the select distribution for the nonterminal $\textrm {VP}$ to depend on the predicate and the second argument of the semantic statement.
With the fully-specified HDPs and their corresponding trees, we have fully specified select. When sampling from $\texttt {select}_{\textrm {A},x}$ for the nonterminal $\textrm {A} \in \mathcal {N}$ and a semantic statement $x \in \mathcal {X}$ , we compute the $m$ semantic features for the given semantic statement: $g_1(x), g_2(x), \hdots , g_m(x)$ . This sequence of indices specifies a path from the root of the tree down to a leaf. We then simply draw a production rule observation from this leaf node, and return the result: $r \sim \text{HDP}(x, T_A) = \texttt {select}_{\textrm {A},x}$ .
There are many other alternatives for defining the select distribution. For instance, a log-linear model can be used to learn dependence on a set of features. The HDP provides statistical advantages, smoothing the learned distributions, resulting in a model more robust to data sparsity issues.
In order to describe inference in this framework, we must define additional concepts and notation. For a nonterminal $\textrm {A}\in \mathcal {N}$ , observe that the paths from the root to the leaves of its HDP tree induce a partition on the set of semantic statements $\mathcal {X}_A$ . More precisely, two semantic statements $x_1,x_2 \in \mathcal {X}_A$ belong to the same equivalence class if they correspond to the same path in an HDP tree.

Question: Do they evaluate the syntactic parses?
Answer: No

Question: What knowledge bases do they use?
Answer: NELL