Common-Knowledge Concept Recognition for SEVA

Abstract: We build a common-knowledge concept recognition system for a Systems Engineer's Virtual Assistant (SEVA) which can be used for downstream tasks such as relation extraction, knowledge graph construction, and question-answering. The problem is formulated as a token classification task similar to named entity extraction. With the help of a domain expert and text processing methods, we construct a dataset annotated at the word-level by carefully defining a labelling scheme to train a sequence model to recognize systems engineering concepts. We use a pre-trained language model and fine-tune it with the labeled dataset of concepts. In addition, we also create some essential datasets for information such as abbreviations and definitions from the systems engineering domain. Finally, we construct a simple knowledge graph using these extracted concepts along with some hyponym relations.

The Systems Engineer's Virtual Assistant (SEVA) BIBREF0 was introduced with the goal to assist systems engineers (SE) in their problem-solving abilities by keeping track of large amounts of information of a NASA-specific project and using the information to answer queries from the user. In this work, we address a system element by constructing a common-knowledge concept recognition system for improving the performance of SEVA, using the static knowledge collected from the Systems Engineering Handbook BIBREF1 that is widely used in projects across the organization as domain-specific commonsense knowledge. At NASA, although there exists knowledge engines and ontologies for the SE domain such as MBSE BIBREF2, IMCE BIBREF3, and OpenCaesar BIBREF4, generic commonsense acquisition is rarely discussed; we aim to address this challenge. SE commonsense comes from years of experience and learning which involves background knowledge that goes beyond any handbook. Although constructing an assistant like SEVA system is the overarching objective, a key problem to first address is to extract elementary common-knowledge concepts using the SE handbook and domain experts. We use the term `common-knowledge' as the `commonsense' knowledge of a specific domain. This knowledge can be seen as a pivot that can be used later to collect `commonsense' knowledge for the SE domain. We propose a preliminary research study that can pave a path towards a comprehensive commonsense knowledge acquisition for an effective Artificial Intelligence (AI) application for the SE domain. Overall structure of this work is summarized in Figure 1. Implementation with demo and dataset is available at: https://github.com/jitinkrishnan/NASA-SE .
Creating commonsense AI still remains an important and challenging task in AI research today. Some of the inspiring works are the CYC project BIBREF5 that tries to serve as a foundational knowledge to all systems with millions of everyday life commonsense assertions, Mosaic Commonsense Knowledge Graphs and Reasoning BIBREF6 that addresses aspects like social situations, mental states, and causal relationships, and Aristo System BIBREF7 that focuses on basic science knowledge. In NASA's context, systems engineering combines several engineering disciplines requiring extreme coordination and is prone to human errors. This, in combination with the lack of efficient knowledge transfer of generic lessons-learned makes most technology-based missions risk-averse. Thus, a comprehensive commonsense engine can significantly enhance the productivity of any mission by letting the experts focus on what they do best.
Concept Recognition (CR) is a task identical to the traditional Named Entity Recognition (NER) problem. A typical NER task seeks to identify entities like name of a person such as `Shakespeare', a geographical location such as `London', or name of an organisation such as `NASA' from unstructured text. A supervised NER dataset consists of the above mentioned entities annotated at the word-token level using labelling schemes such as BIO which provides beginning (B), continuation or inside (I), and outside (O) representation for each word of an entity. BIBREF8 is the current top-performing NER model for CoNLL-2003 shared task BIBREF9. Off-the-shelf named entity extractors do not suffice in the SE common-knowledge scenario because the entities we want to extract are domain-specific concepts such as `system architecture' or `functional requirements' rather than physical entities such as `Shakespeare' or `London'. This requires defining new labels and fine-tuning.
Relation extraction tasks extract semantic relationships from text. These extractors aim to connect named entities such as `Shakespeare' and `England' using relations such as `born-in'. Relations can be as simple as using hand-built patterns or as challenging as using unsupervised methods like Open IE BIBREF10; with bootstrapping, supervised, and semi-supervised methods in between. BIBREF11 and BIBREF12 are some of the high performing models that extract relations from New York Times Corpus BIBREF13 and TACRED challenges BIBREF14 respectively. Hyponyms represent hierarchical connection between entities of a domain and represent important relationships. For instance, a well-known work by BIBREF15 uses syntactic patterns such as [Y such as A, B, C], [Y including X], or [Y, including X] to extract hyponyms. Our goal is to extract preliminary hyponym relations from the concepts extracted by the CR and to connect the entities through verb phrases.
SE concepts are less ambiguous as compared to generic natural language text. A word usually means one concept. For example, the word `system' usually means the same when referring to a `complex system', `system structure', or `management system' in the SE domain. In generic text, the meaning of terms like `evaluation', `requirement', or `analysis' may contextually differ. We would like domain specific phrases such as `system evaluation', `performance requirement', or `system analysis' to be single entities. Based on the operational and system concepts described in BIBREF0, we carefully construct a set of concept-labels for the SE handbook which is shown in the next section.
abb: represents abbreviations such as TRL representing Technology Readiness Level.
grp: represents a group of people or an individual such as Electrical Engineers, Systems Engineers or a Project Manager.
syscon: represents any system concepts such as engineering unit, product, hardware, software, etc. They mostly represent physical concepts.
opcon: represents operational concepts such as decision analysis process, technology maturity assessment, system requirements review, etc.
seterm: represents generic terms that are frequently used in SE text and those that do not fall under syscon or opcon such as project, mission, key performance parameter, audit etc.
event: represents event-like information in SE text such as Pre-Phase A, Phase A, Phase B, etc.
org: represents an organization such as `NASA', `aerospace industry', etc.
art: represents names of artifacts or instruments such as `AS1300'
cardinal: represents numerical values such as `1', `100', 'one' etc.
loc: represents location-like entities such as component facilities or centralized facility.
mea: represents measures, features, or behaviors such as cost, risk, or feasibility.
Abbreviations are used frequently in SE text. We automatically extract abbreviations using simple pattern-matching around parentheses. Given below is a sample regex that matches most abbreviations in the SE handbook.
r"\([ ]*[A-Z][A-Za-z]*[ ]*\)"
An iterative regex matching procedure using this pattern over the preceding words will produce the full phrase of the abbreviation. `A process to determine a system’s technological maturity based on Technology Readiness Levels (TRLs)' produces the abbreviation TRL which stands for Technology Readiness Levels. `Define one or more initial Concept of Operations (ConOps) scenarios' produces the abbreviation ConOps which stands for Concept of Operations. We pre-label these abbreviations as concept entities. Many of these abbreviations are also provided in the Appendix section of the handbook which is also extracted and used as concepts.
Various locations of the handbook and the glossary provide definitions of several SE concepts. We collect these and compile a comprehensive definitions document which is also used for the concept recognition task. An example definition and its description is shown below:
Definition: Acceptable Risk
Description: The risk that is understood and agreed to by the program/project, governing authority, mission directorate, and other customer(s) such that no further specific mitigating action is required.
Using python tools such as PyPDF2, NLTK, and RegEx we build a pipeline to convert PDF to raw text along with extensive pre-processing which includes joining sentences that are split, removing URLs, shortening duplicate non-alpha characters, and replacing full forms of abbreviations with their shortened forms. We assume that the SE text is free of spelling errors. For the CR dataset, we select coherent paragraphs and full sentences by avoiding headers and short blurbs. Using domain keywords and a domain expert, we annotate roughly 3700 sentences at the word-token level. An example is shown in Figure 2 and the unique tag count is shown in Table 1.
Any language model can be used for the purpose of customizing an NER problem to CR. We choose to go with BERT BIBREF16 because of its general-purpose nature and usage of contextualized word embeddings.
In the hand-labelled dataset, each word gets a label. The idea is to perform multi-class classification using BERT's pre-trained cased language model. We use pytorch transformers and hugging face as per the tutorial by BIBREF17 which uses $BertForTokenClassification$. The text is embedded as tokens and masks with a maximum token length. This embedded tokens are provided as the input to the pre-trained BERT model for a full fine-tuning. The model gives an F1-score of $0.89$ for the concept recognition task. An 80-20 data split is used for training and evaluation. Detailed performance of the CR is shown in Table 2 and 3. Additionally, we also implemented CR using spaCy BIBREF18 which also produced similar results.
In this work, for relation extraction, we focus on hyponyms and verb phrase chunking. Hyponyms are more specific concepts such as earth to planet or rose to flower. Verb phrase chunking connects the named entities recognized by the CR model through verbs.
The definition document consists of 241 SE definitions and their descriptions. We iteratively construct entities in increasing order of number of words in the definitions with the help of their parts-of-speech tags. This helps in creating subset-of relation between a lower-word entity and a higher-word entity. Each root entity is lemmatized such that entities like processes and process appear only once.
Using the words (especially nouns) that surround an already identified named entity, more specific entities can be identified. This is performed on a few selected entity tags such as opcon and syscon. For example, consider the sentence `SE functions should be performed'. `SE' has tag NNP and `functions' has tag NNS. We create a relation called subset-of between `SE functions' and `SE'.
Relations from abbreviations are simple direct connections between the abbreviation and its full form described in the abbreviations dataset. Figure FIGREF25 shows a snippet of knowledge graph constructed using stands-for and subset-of relationships. Larger graphs are shown in the demo.
Finally, we explore creating contextual triples from sentences using all the entities extracted using the CR model and entities from definitions. Only those phrases that connect two entities are selected for verb phrase extraction. Using NLTK's regex parser and chunker, a grammar such as
VP: {(<MD>|<R.*>|<I.*>|<VB.*>|<JJ.*>|
<TO>)*<VB.*>+(<MD>|<R.*>|<I.*>|<VB.*>|
<JJ.*>|<TO>)*}
with at least one verb, can extract relation-like phrases from the phrase that links two concepts. An example is shown in Figure FIGREF27. Further investigation of relation extraction from SE handbook is left as future work.
We presented a common-knowledge concept extractor for the Systems Engineer's Virtual Assistant (SEVA) system and showed how it can be beneficial for downstream tasks such as relation extraction and knowledge graph construction. We construct a word-level annotated dataset with the help of a domain expert by carefully defining a labelling scheme to train a sequence labelling task to recognize SE concepts. Further, we also construct some essential datasets from the SE domain which can be used for future research. Future directions include constructing a comprehensive common-knowledge relation extractor from SE handbook and incorporating such human knowledge into a more comprehensive machine-processable commonsense knowledge base for the SE domain.

Figure and table captions:
Figure 1: Common-knowledge concept recognition and simple relation extraction
Table 2: Performance of different labels
Table 1: Unique Tag Count from the CR dataset
Figure 2: A Snippet of the concept-labelled dataset
Table 3: Overall Performance of CR; For fairness, we also provide the accuracy when the most common ‘O’-tag is excluded from the analysis.
Figure 3: A snippet of the knowledge graph generated
Figure 4: Relation Extraction using Verb Phrase

Questions about this paper:
How many domain experts were involved into creation of dataset?
What metrics are used for evaluation?
How does labeling scheme look like?

Answers:
One domain expert.
Precision, recall, f1-score, and support.
1. abb
2. grp
3. syscon
4. opcon
5. seterm
6. event
7. org
8. art
9. cardinal
10. loc
11. mea