Sentence processing in spiking neurons: A biologically plausible left-corner parser

Sentence processing in spiking neurons: A biologically plausible left-corner parser Terrence C. Stewart (tcstewar@uwaterloo.ca) Xuan Choo (fchoo@uwaterloo.ca) Chris Eliasmith (celiasmith@uwaterloo.ca) Center for Theoretical Neuroscience, University of Waterloo Waterloo, ON, Canada N2L 3G1 Abstract A long-standing challenge in cognitive science is how neurons could be capable of the flexible structured processing that is the hallmark of cognition. We present a spiking neural model that can be given an input sequence of words (a sentence) and produces a structured tree-like representation indicating the parts of speech it has identified and their relations to each other. While this system is based on a standard left-corner parser for constituency grammars, the neural nature of the model leads to new capabilities not seen in classical implementations. For example, the model gracefully decays in performance as the sentence structure gets larger. Unlike previous attempts at building neural parsing systems, this model is highly robust to neural damage, can be applied to any binary-constituency grammar, and requires relatively few neurons (~150,000). Keywords: Neural engineering framework; vector symbolic architectures; left-corner parsing; syntax; binary trees; computational neuroscience Introduction Human language processing requires not only the ability to represent complex structures, but also the ability to create these representations out of a serial sequence of words. This system is flexible enough to work for a huge variety of possible sentences, including (crucially), novel sentences. Modern linguistics is founded on the principle that these structured representations are tree-like, and that this tree structure imposes useful order on the sentence. For example, “the dog ran” can be parsed as follows: Here the sentence (S) is divided into a noun phrase (NP) and a verb phrase (VP). The noun phrase is divided into a determiner (DT) “the” and a noun (N) “dog”. The verb phrase consists of a single verb (V) “ran”. To describe the space of possible trees we define constituency rules indicating how the parts can fit together. For example, the above sentence is consistent with the following constituency grammar rules: S → [NP VP] DT → “the” NP → [DT N] N → “dog” VP → [V] V → “ran” (Structural rules) (Vocabulary) By adding more structural rules (on the left), we can build more complex sentences. By adding more words (on the right), we can increase our vocabulary. Left-Corner Parsing As more rules are included in the grammar, parsing becomes more complicated. More rules mean more possibilities, and finding a tree that is consistent with the rules can become computationally expensive. For example, there may be the two rules VP → [V] and VP → [V NP], leading to an explosion of possible structures to search through. Furthermore, a single word may have multiple interpretations, such as N → “dog” and V → “dog”. The Left-Corner parsing algorithm addresses this problem by combining bottom-up and top-down information. The top-down information is what part of speech we are currently looking for (a sentence, a noun phrase, a determiner, etc.). The bottom-up information is the single word we are currently processing. So, if we are currently looking for a noun, we will first try interpreting the word “dog” as a noun, rather than a verb. If this does not lead to a successful parse, then the algorithm will backtrack to this point and try the other option. This approach drastically reduces the amount of backtracking needed. For “the dog ran”, the algorithm proceeds as follows: • Top-down: look for S • Bottom-up: see “the” • Apply rule DT → “the” • Apply rule NP → [DT N] • Store the partially completed tree ◦ Top-down: look for N for previous rule ◦ Bottom-up: see “dog” ◦ Apply rule N → “dog” ◦ Merge with previously stored tree • Apply rule S → [NP VP] • Store the partially completed tree ◦ Top-down: look for VP for previous rule ◦ Bottom-up: see “ran” ◦ Apply rule V → “ran” ◦ Apply rule VP → [V] ◦ Merge with previously stored tree As has been pointed out (e.g. Johnson-Laird, 1983), this algorithm matches well with observed human sentence comprehension. For example, it has difficulty with “garden- path” sentences such as the classic “the horse raced past the barn fell”, which is difficult for most people to interpret even though it has a similar form as the easier sentence “the deer shot by the hunter fell”. A left-corner parser has the same tendency as people do to connect the ambiguous word “raced” as part of a S → [NP VP] rule.