Writing Linguistic Rules for Natural Language Processing | by Jenny Lee | Towards Data Science

Ruminations of a LinguistA solution: Rule-Based Syntactic Feature ExtractionAn example: Question Type Extraction Using spaCy
Ruminations of a Linguist
A solution: Rule-Based Syntactic Feature Extraction
An example: Question Type Extraction Using spaCy
When I first started exploring data science towards the end of my Ph.D. program in linguistics, I was pleased to discover the role of linguistics — specifically, linguistic features — in the development of NLP models. At the same time, I was a bit perplexed by why there was relatively little talk of exploiting syntactic features (e.g., level of clausal embedding, presence of coordination, type of speech act, etc.) compared to other types of features, such as how many times a certain word occurs in the text (lexical features), word similarity measures (semantic features), and even where in the document a word or phrase occurs (positional features).
For example, in a sentiment analysis task, we may use a list of content words (adjectives, nouns, verbs, and adverbs) as features to predict the semantic orientation of user feedback (i.e., positive or negative). In another feedback classification task, we may curate domain-specific lists of words or phrases to train a model that can direct user comments to an appropriate division of support, e.g., billing, technical, or customer service.
Both these tasks require pre-built lexicons for feature extraction, and there are many such lexicons out there for public use, such as SocialSent and these ones curated by a Medium writer. (Also check out this research article which discusses different methods of building sentiment lexicons). While lexical features have played an important role in the development of many NLP applications, there has been relatively little work involving the use of syntactic features (one big exception being perhaps in the domain of grammar error detection).
This was baffling (and a little bit irksome) to me as someone coming from a tradition of linguistics that gives syntax something of “supremacy” over semantics and phonology in the general architecture of grammar. This, of course, isn’t to say that linguists perceive syntax to be in some ways more important than the other branches of linguistics — but simply that it occupies a position of primacy in many models of grammar that posit that syntactic form is derived before a semantic or phonetic interpretation is achieved. Accustomed to this “centrality” of the syntactic module in linguistic theory, I struggled to make sense of the lukewarm attitude toward syntactic structure in NLP, which sharply contrasted with the great confidence that NLP developers put in lexical content as features.
Language, as imagined by NLP, made me think of a fragmented Mr. Potato Head at times — two-dimensional body parts strewn on a page of a coloring book, words without relations to each other.
Some of the unease I felt surrounding this apparent oversimplification of language (as a bag of words in my earliest understanding of it) did eventually dissipate when I caught on to the “magic” of machine learning/deep learning: Even without explicit coding, “it learns the syntax, too.” (And ironically, that is also how a child acquires language — without explicit instruction.)
As a linguist (specifically a morphosyntactician), though, I am still honestly a bit hung up on the reality that syntactic features have not garnered as much attention as lexical features in NLP/ML, and I am inclined to attribute this asymmetry to the following reasons:
As students of NLP, we are repeatedly introduced to an NLP pipeline that includes preprocessing techniques like stop words removal, lemmatization/stemming, and bag-of-words representation. These techniques are valuable in many cases but are inherently subtractive: They remove chunks of syntactic (and/or morphological) information that are present in the original text. Stop words removal eliminates function words which encode information about the relations between constituents/phrases, e.g., into, from, to, the, and, etc. Lemmatization and stemming removes important inflectional features such as tense and agreement as well as derivational affixes that may be crucial to determining the precise meaning of a word (e.g., -er in teacher, -ment in -embezzlement). And of course, bag-of-words representations disregard word order, a significant loss for a language like English which does not use morphological case to indicate word order. When these steps become normal and recurring parts of our everyday NLP pipeline, it has the potential to obscure or minimize the role that syntactic structure actually does play in natural language processing, consequently hindering or discouraging any efforts to systematically measure its impact on NLP models.
I also speculate that the relative obscurity of syntactic features is in large part due to the practical challenges of writing sophisticated enough rules. NLP experts come into the field with varying degrees of knowledge in syntax, and without a firm grasp of the fundamentals and universals of syntactic structure that underlie every language as well as, to some extent, constraints on cross-linguistic variation, it may be difficult to know what syntactic features are relevant for a particular language, let alone go about writing heuristics to extract them. Unlike lexical features which in many cases amount to simple word counts, crafting well-thought-out linguistic rules at the sentence (syntactic) level is a much more challenging task that requires considerable domain knowledge.
I also came to see that some of our most popular and commercialized NLP products do not require very complex syntactic representations of input text or speech — making syntactic features less than indispensable in the development of the models that power them. I think the reasons are (1) human speech in machine-human interaction tends to be oversimplified to begin with (“Hey Google, SF weather” vs. “Hey John, would you mind telling me what the weather is in San Francisco today?”) and (2) simple parses are more amenable to less computationally expensive systems than complex parses, so we’re compelled to work with those constraints.
To be clear, the three things I outlined above are challenges/hindrances to using syntactic features, not justifications for excluding them, in feature engineering. While I’m not sure how much syntactic features can contribute to the performance of a model compared to other features, I do believe that as we seek to build machines that can interact with humans in much more subtle, natural, and sophisticated ways, syntactic input and representation will play an increasingly prominent role in the future of NLP. I think we would benefit greatly by doing these three things in response:
Experiment more with syntactic features in our everyday ML endeavors and invest in streamlining that process so it becomes a repeated part of the pipeline.Create more open-source code for linguistic feature extraction.Hire more linguists — and help them grow in technical skills if necessary. Invest in them for the unique return they will bring.
Experiment more with syntactic features in our everyday ML endeavors and invest in streamlining that process so it becomes a repeated part of the pipeline.
Create more open-source code for linguistic feature extraction.
Hire more linguists — and help them grow in technical skills if necessary. Invest in them for the unique return they will bring.
In all three of these tasks, I think we would get the most mileage by exploiting, specifically, a rule-driven system. Syntax is an area that requires precise knowledge about a specific language and therefore lends itself well to rule-based representation (in NLP, that is), perhaps more so than any other area of grammar. The proposal to take a rule-based approach competes with taking an entirely ML approach to build feature sets. In the latter, you train a model that predicts the value of a specific feature to feed into a model that predicts the output of some target variable. This process is likely to be very time-consuming and expensive, however, because you’re training a model to train a model. And that is why I would recommend using rules as part of a hybrid system in almost all cases. Hybrid approaches combine an ML algorithm-driven base model with a rule-based system that serves as a kind of post-processing filter, and they are old news. There have been many successful applications of them, like this one and at my own job.
Another advantage of rules and rule-based systems is that they can be used flexibly in a variety of ways and therefore bring value and rigor to almost every stage of the ML pipeline. In addition to using them to extract features during the feature engineering stage, they can also be used to:
Pre-label large datasets to create noisy training data during data preparation (a labeling function)Exclude certain annotations during data preprocessing (a pre-filter)Filter out false positives from a classifier’s output before/during model evaluation (a post-filter)
Pre-label large datasets to create noisy training data during data preparation (a labeling function)
Exclude certain annotations during data preprocessing (a pre-filter)
Filter out false positives from a classifier’s output before/during model evaluation (a post-filter)
Therefore, it is hardly appropriate to pit rules or rule-based systems against ML algorithms as some might be inclined to do. Rather, we must acknowledge that they complement each other in some very compelling ways, adding a serious edge to the overall architecture.
In this final section, I will provide a practical guide to writing some syntactic rules for feature extraction. I will use spaCy to do this, focusing on extracting question type. I chose question type because this category can illustrate particularly well the incredible value of linguistic knowledge in building a partially rule-driven NLP model. Moreover, there are probably many use cases for question type extraction (and by extension, sentence type), so hopefully, someone will benefit from the code below. Some applications I can think of are: a fine-grained classification of questions (is the sentence a wh-question, a yes or no question, or a tag question?), speech act detection (is the comment a question, request, or demand?), some comparative speech analyses (which individual or group uses more politeness strategies in their speech?), etc.
With this concluding section, I ultimately hope to have driven home both the relevance of linguistic knowledge for NLP developers and the relevance of rules or rule-based systems in NLP/ML tasks.
In writing linguistic rules, I generally follow these six steps:
Identify categories of interest (e.g., “wh-question”, “polar question”, “not a question”).Come up with one or two big linguistic generalizations about each category.Come up with some counterexamples to the generalizations and revise/expand them as necessary.Write rules capturing the new generalizations.Test the rules on a bunch of examples.Fine-tune the rules by addressing any false positives and testing new examples.
Identify categories of interest (e.g., “wh-question”, “polar question”, “not a question”).
Come up with one or two big linguistic generalizations about each category.
Come up with some counterexamples to the generalizations and revise/expand them as necessary.
Write rules capturing the new generalizations.
Test the rules on a bunch of examples.
Fine-tune the rules by addressing any false positives and testing new examples.
Often these steps will not be perfectly sequential. For example, identifying the categories you want to target (step 1) basically amounts to stating some generalizations about them (step 2), so you may do these steps at the same time. Before I illustrate how this flow can guide our rule writing, let’s dig into spaCy a bit.
spaCy is a high-performance Python library that boasts of numerous features and capabilities to help you build your favorite NLP application, including tokenization, named entity recognition, part-of-speech tagging, dependency parsing, pre-trained word vectors and similarity, and built-in visualizers. Check out its full suite of features here, but here we’ll only be concerned with two of its processing components tbat will enable us to represent the syntactic structure of any sentence, namely, tagger and parser. The basic flow of using spaCy goes like this: First, load a model of your choice (e.g., "en_core_web_sm"). You can then examine the pipeline components by calling pipeline or pipeline_name on nlp.
Next, call nlp by passing in a string and converting it to a spaCy Doc, a sequence of tokens (line 2 below). The syntactic structure of Doc can be constructed by accessing the following attributes of its component tokens:
pos_: The coarse-grained part-of-speech tag.
tag_: The fine-grained part-of-speech tag.
dep_: The syntactic dependency tag, i.e. the relation between tokens.
head: The syntactic governor, or the immediately dominating token
I find it helpful to print out the values of these attributes along with token names and indices in tuples, like this:
If you’d like to visualize the parse, type your sentence into spaCy’s built-in visualizer displaCy:
The arrows indicate the head-dependent relations between tokens and each token is labeled with a coarse-level POS tag (fine-level (tag_) is not shown here).
These two steps often go hand in hand. In coming up with rules for question type extraction, we must recognize that there are two types of questions in any language: wh-questions and polar questions. Read these generalizations carefully:
Generalizations about question types:(1) Wh-questions (aka content questions): start with an interrogative word (a “wh-phrase”) like who, what, where, and how, and call for specific information.(2) Polar questions (aka yes or no questions): can be answered in the affirmative or the negative (with a yes or a no response), and begin with an auxiliary (did, do) or modal (will, can) verb.
You may find it helpful to visualize these question types using displaCy. The last screenshot contains a parse for a wh-question and here’s one for a polar question:
In addition to a visual parse, I like being able to see the pos-tag-dep triplet for each token, like this:
Using your knowledge of Penn Treebank annotation conventions (because that’s what spaCy uses), you can try writing v1 functions that would capture the generalizations above. I usually prefer to write separate functions for the different categories rather than writing a single function that contains complicated if/then logic. Here’s a quick first version attempting to capture the generalizations in (1)-(2).
Coming up with counterexamples to your initial generalizations is essential because they will challenge your basic assumptions about the linguistic phenomenon under consideration (here, question formation) and force you to identify a common thread between the different examples to express their behavior and characteristics in a unified, generalized way. Ultimately, that will help help you write simpler but more robust and comprehensive rules.
The two generalizations given above are reproduced here to save you scrolling:
(1) Wh-questions (aka content questions): start with an interrogative word (a “wh-phrase”) like who, what, where, and how, and call for specific information.(2) Polar questions (aka yes or no questions): can be answered in the affirmative or the negative (with a yes or a no response), and begin with an auxiliary (did, do) or modal (will, can) verb.
Here’s the first pair of counterexamples to the generalization in (1):
(3) What you say is unbelievable.(4) Who you're going to kill is not my problem.
We see that (1) is not borne out by these sentences because despite beginning with a wh-word, they are not wh-questions! They are what linguists call wh-clefts, or pseudoclefts. Pseudoclefts are often described as giving emphasis, or “focus”, to part of the sentence and have the form in (5):
(5) free relative + "be" + focused constituent
A free relative is a wh-relative clause that (apparently) lacks a head noun (so “what you say” instead of “the thing you say”). This part of the pseudocleft introduces a topic (what the sentence is about) and is linked to a focused constituent which introduces new information (“unbelievable”) by the verb “be”.
Here are some additional counterexamples to (1):
(5) In which article did they talk about spaCy?(6) To whom did you read the article?
These sentences fail to bear out (1) for a different reason: While they don’t start with a wh-phrase as the generalization says, they still are wh-questions in that they can be answered with specific information (e.g., an answer to (6) can be “I read the article to my sister.”). These sentences are so-called pied-piped constructions and are characterized by a wh-word appearing along with a preposition, as though it has “dragged” the other to the front of the sentence. (6) in particular can be contrasted with another sentence that “strands” a preposition: “Who did you read the article to?”.
Finally, here are two counterexamples to the generalization about polar questions, stated in (2):
(7) Is it going to rain tonight?(8) Were you sad about her leaving?
These sentences show that in addition to auxiliary and modal verbs, the copula (“be”) as a main verb (vs. an auxiliary) should be included in the category of verbs that can appear at the beginning of polar questions. Stated more simply, a verb and a subject undergo inversion in polar questions when the verb is an auxiliary, a modal, or a form of “be” that serves as the main verb. This brings us to the following revised generalizations:
(1') Wh-questions (aka content questions): contain an interrogative word (a “wh-phrase”) like who, what, where, and how, and call for more specific information.(2') Polar questions (aka yes or no questions): can be answered in the affirmative or the negative (with a yes or a no response), and display subject-verb inversion where the verb must be either an auxiliary (did, do), a modal (will, can), or a form of the main verb be.
I think this step demonstrates quite palpably that linguistic knowledge is indispensable to writing sophisticated feature-extracting rules. Knowing that English employs various strategies to signal interrogative sentences such as subject-verb inversion and wh-movement, as well as recognizing (and having names for) interesting linguistic phenomena, like pied-piping and clefting, can help you formulate generalizations that can in turn help you write comprehensive rules that are not overly specific.
We can now modify the earlier versions of the functions, is_wh_question_v1 and is_polar_question_v1, to conform to the new generalizations (1') and (2'). This part will require some time and practice before we can feel very confident and comfortable, so be patient! Below I’ll highlight a few core things that characterize each of the v2 functions:
In is_wh_question_v2:
Referring to spaCy tags (e.g., "WDT", "WP", etc.) makes your rules more general than referring to the individual wh-words (line 3).
Pied-piping can be captured by a prep dependency relation between the wh-word and its head (the syntactic parent, or “governor” of the token) (line 8).
Pseudoclefts are excluded by returning False if the wh-word stands in a csubj or advcl relation to its head (lines 11–13).
In is_polar_question_v2, we need to account for two situations:
The question occurs in a non-copular construction in which “is” (or another form of “be”, like “were”) functions as an auxiliary, e.g., “Is she using spaCy?” (the main verb here is “using”.)The question occurs in a copular construction in which “is” functions as the main verb, “Is the mouse dead?”
The question occurs in a non-copular construction in which “is” (or another form of “be”, like “were”) functions as an auxiliary, e.g., “Is she using spaCy?” (the main verb here is “using”.)
The question occurs in a copular construction in which “is” functions as the main verb, “Is the mouse dead?”
Type 1 is captured in lines 14–16, and the inversion of the subject is with respect to the auxiliary. (The same lines of code can also account for subject-modal inversion, like “Can you read the article?”).
Type II is captured in lines 20–22, and the inversion of the subject is with respect to the root, i.e., the sole token whose dependency tag is "ROOT" in the doc.
The mutual exclusivity of polar questions and wh-questions is captured in lines 9–10.
Finally, to put all of the above together, we can write a test function that takes a list of example sentences and prints the question type for each sentence.
When I run get_question_type(sentences), I get the following output:
In [352]: get_question_type(sentences)Is that for real? -- polarCan you stop? -- polarDo you love John? -- polarAre you sad? -- polarWas she singing? -- polarWon't you come over for dinner? -- polarWould you help me? -- polarWho do you love? -- whWhose child is that. -- whHow do you know him? -- whTo whom did she read the book? -- whI'm hungry. -- not a questionSpacy is so fun! -- not a questionTell me what you mean. -- not a questionWould love to help you. -- not a questionDon't be sad. -- not a questionWhatever you want. -- wh  # false positive What you say is impossible. -- not a questionWhere you go, I will go. -- not a question
Everything is correct except for the third to last sentence, “Whatever you want.”. With the v2 functions, we have achieved an accuracy of 18/19 = 94.7%. Not bad! Note that these rules are truly sensitive to the underlying syntactic structure of each sentence and not to some superficial properties like the presence of a question mark (as illustrated by the correct classification of the example “Whose child is that.”).
And for good measure:
from sklearn.metrics import classification_reportcr = classification_report(true_values, rules_predictions)print(cr)                precision    recall  f1-score   supportnot a question       1.00      0.88      0.93         8         polar       1.00      1.00      1.00         7            wh       0.80      1.00      0.89         4
As a final step, you will want to fine-tune your rules by addressing any false positives to increase precision (e.g., “Whatever you want.” was incorrectly classified as a wh-question) and testing with more examples (and more complex examples) to increase recall. When you are satisfied, apply the rules to new data and use the results at any stage of your workflow that you deem appropriate. For example, you may choose to incorporate them as features during feature engineering or use them as early as the data pipeline to generate noisy data for annotation.
Linguistic rule writing for NLP/machine learning is a rich, iterative process that requires a deep understanding of language and the ability to codify that knowledge in the form of general heuristics in a programming language. While rules are probably not adequate to stand alone in most applications of NLP and good rule writing almost always requires expert knowledge, they have many advantages and are a great complement to ML algorithms. If I have convinced you at all of their value, please give spaCy a try and feel free to suggest a v3 function below or add a request for rules for other features. :)