INTRODUCTION
The Systems Engineer's Virtual Assistant (SEVA) BIBREF0 was introduced with the goal to assist systems engineers (SE) in their problem-solving abilities by keeping track of large amounts of information of a NASA-specific project and using the information to answer queries from the user. In this work, we address a system element by constructing a common-knowledge concept recognition system for improving the performance of SEVA, using the static knowledge collected from the Systems Engineering Handbook BIBREF1 that is widely used in projects across the organization as domain-specific commonsense knowledge. At NASA, although there exists knowledge engines and ontologies for the SE domain such as MBSE BIBREF2, IMCE BIBREF3, and OpenCaesar BIBREF4, generic commonsense acquisition is rarely discussed; we aim to address this challenge. SE commonsense comes from years of experience and learning which involves background knowledge that goes beyond any handbook. Although constructing an assistant like SEVA system is the overarching objective, a key problem to first address is to extract elementary common-knowledge concepts using the SE handbook and domain experts. We use the term `common-knowledge' as the `commonsense' knowledge of a specific domain. This knowledge can be seen as a pivot that can be used later to collect `commonsense' knowledge for the SE domain. We propose a preliminary research study that can pave a path towards a comprehensive commonsense knowledge acquisition for an effective Artificial Intelligence (AI) application for the SE domain. Overall structure of this work is summarized in Figure 1. Implementation with demo and dataset is available at: https://github.com/jitinkrishnan/NASA-SE .
BACKGROUND AND MOTIVATION
Creating commonsense AI still remains an important and challenging task in AI research today. Some of the inspiring works are the CYC project BIBREF5 that tries to serve as a foundational knowledge to all systems with millions of everyday life commonsense assertions, Mosaic Commonsense Knowledge Graphs and Reasoning BIBREF6 that addresses aspects like social situations, mental states, and causal relationships, and Aristo System BIBREF7 that focuses on basic science knowledge. In NASA's context, systems engineering combines several engineering disciplines requiring extreme coordination and is prone to human errors. This, in combination with the lack of efficient knowledge transfer of generic lessons-learned makes most technology-based missions risk-averse. Thus, a comprehensive commonsense engine can significantly enhance the productivity of any mission by letting the experts focus on what they do best.
Concept Recognition (CR) is a task identical to the traditional Named Entity Recognition (NER) problem. A typical NER task seeks to identify entities like name of a person such as `Shakespeare', a geographical location such as `London', or name of an organisation such as `NASA' from unstructured text. A supervised NER dataset consists of the above mentioned entities annotated at the word-token level using labelling schemes such as BIO which provides beginning (B), continuation or inside (I), and outside (O) representation for each word of an entity. BIBREF8 is the current top-performing NER model for CoNLL-2003 shared task BIBREF9. Off-the-shelf named entity extractors do not suffice in the SE common-knowledge scenario because the entities we want to extract are domain-specific concepts such as `system architecture' or `functional requirements' rather than physical entities such as `Shakespeare' or `London'. This requires defining new labels and fine-tuning.
Relation extraction tasks extract semantic relationships from text. These extractors aim to connect named entities such as `Shakespeare' and `England' using relations such as `born-in'. Relations can be as simple as using hand-built patterns or as challenging as using unsupervised methods like Open IE BIBREF10; with bootstrapping, supervised, and semi-supervised methods in between. BIBREF11 and BIBREF12 are some of the high performing models that extract relations from New York Times Corpus BIBREF13 and TACRED challenges BIBREF14 respectively. Hyponyms represent hierarchical connection between entities of a domain and represent important relationships. For instance, a well-known work by BIBREF15 uses syntactic patterns such as [Y such as A, B, C], [Y including X], or [Y, including X] to extract hyponyms. Our goal is to extract preliminary hyponym relations from the concepts extracted by the CR and to connect the entities through verb phrases.
CONCEPT RECOGNITION
SE concepts are less ambiguous as compared to generic natural language text. A word usually means one concept. For example, the word `system' usually means the same when referring to a `complex system', `system structure', or `management system' in the SE domain. In generic text, the meaning of terms like `evaluation', `requirement', or `analysis' may contextually differ. We would like domain specific phrases such as `system evaluation', `performance requirement', or `system analysis' to be single entities. Based on the operational and system concepts described in BIBREF0, we carefully construct a set of concept-labels for the SE handbook which is shown in the next section.
CONCEPT RECOGNITION ::: BIO Labelling Scheme
abb: represents abbreviations such as TRL representing Technology Readiness Level.
grp: represents a group of people or an individual such as Electrical Engineers, Systems Engineers or a Project Manager.
syscon: represents any system concepts such as engineering unit, product, hardware, software, etc. They mostly represent physical concepts.
opcon: represents operational concepts such as decision analysis process, technology maturity assessment, system requirements review, etc.
seterm: represents generic terms that are frequently used in SE text and those that do not fall under syscon or opcon such as project, mission, key performance parameter, audit etc.
event: represents event-like information in SE text such as Pre-Phase A, Phase A, Phase B, etc.
org: represents an organization such as `NASA', `aerospace industry', etc.
art: represents names of artifacts or instruments such as `AS1300'
cardinal: represents numerical values such as `1', `100', 'one' etc.
loc: represents location-like entities such as component facilities or centralized facility.
mea: represents measures, features, or behaviors such as cost, risk, or feasibility.
CONCEPT RECOGNITION ::: Abbreviations
Abbreviations are used frequently in SE text. We automatically extract abbreviations using simple pattern-matching around parentheses. Given below is a sample regex that matches most abbreviations in the SE handbook.
r"\([ ]*[A-Z][A-Za-z]*[ ]*\)"
An iterative regex matching procedure using this pattern over the preceding words will produce the full phrase of the abbreviation. `A process to determine a system’s technological maturity based on Technology Readiness Levels (TRLs)' produces the abbreviation TRL which stands for Technology Readiness Levels. `Define one or more initial Concept of Operations (ConOps) scenarios' produces the abbreviation ConOps which stands for Concept of Operations. We pre-label these abbreviations as concept entities. Many of these abbreviations are also provided in the Appendix section of the handbook which is also extracted and used as concepts.
CONCEPT RECOGNITION ::: Common-Knowledge Definitions
Various locations of the handbook and the glossary provide definitions of several SE concepts. We collect these and compile a comprehensive definitions document which is also used for the concept recognition task. An example definition and its description is shown below:
Definition: Acceptable Risk
Description: The risk that is understood and agreed to by the program/project, governing authority, mission directorate, and other customer(s) such that no further specific mitigating action is required.
CONCEPT RECOGNITION ::: CR Dataset Construction and Pre-processing
Using python tools such as PyPDF2, NLTK, and RegEx we build a pipeline to convert PDF to raw text along with extensive pre-processing which includes joining sentences that are split, removing URLs, shortening duplicate non-alpha characters, and replacing full forms of abbreviations with their shortened forms. We assume that the SE text is free of spelling errors. For the CR dataset, we select coherent paragraphs and full sentences by avoiding headers and short blurbs. Using domain keywords and a domain expert, we annotate roughly 3700 sentences at the word-token level. An example is shown in Figure 2 and the unique tag count is shown in Table 1.
CONCEPT RECOGNITION ::: Fine tuning with BERT
Any language model can be used for the purpose of customizing an NER problem to CR. We choose to go with BERT BIBREF16 because of its general-purpose nature and usage of contextualized word embeddings.
In the hand-labelled dataset, each word gets a label. The idea is to perform multi-class classification using BERT's pre-trained cased language model. We use pytorch transformers and hugging face as per the tutorial by BIBREF17 which uses $BertForTokenClassification$. The text is embedded as tokens and masks with a maximum token length. This embedded tokens are provided as the input to the pre-trained BERT model for a full fine-tuning. The model gives an F1-score of $0.89$ for the concept recognition task. An 80-20 data split is used for training and evaluation. Detailed performance of the CR is shown in Table 2 and 3. Additionally, we also implemented CR using spaCy BIBREF18 which also produced similar results.
RELATION EXTRACTION
In this work, for relation extraction, we focus on hyponyms and verb phrase chunking. Hyponyms are more specific concepts such as earth to planet or rose to flower. Verb phrase chunking connects the named entities recognized by the CR model through verbs.
RELATION EXTRACTION ::: Hyponyms from Definitions
The definition document consists of 241 SE definitions and their descriptions. We iteratively construct entities in increasing order of number of words in the definitions with the help of their parts-of-speech tags. This helps in creating subset-of relation between a lower-word entity and a higher-word entity. Each root entity is lemmatized such that entities like processes and process appear only once.
RELATION EXTRACTION ::: Hyponyms from POS tags
Using the words (especially nouns) that surround an already identified named entity, more specific entities can be identified. This is performed on a few selected entity tags such as opcon and syscon. For example, consider the sentence `SE functions should be performed'. `SE' has tag NNP and `functions' has tag NNS. We create a relation called subset-of between `SE functions' and `SE'.
RELATION EXTRACTION ::: Relations from Abbreviations
Relations from abbreviations are simple direct connections between the abbreviation and its full form described in the abbreviations dataset. Figure FIGREF25 shows a snippet of knowledge graph constructed using stands-for and subset-of relationships. Larger graphs are shown in the demo.
RELATION EXTRACTION ::: Relation Extraction using Verb Phrase Chunking
Finally, we explore creating contextual triples from sentences using all the entities extracted using the CR model and entities from definitions. Only those phrases that connect two entities are selected for verb phrase extraction. Using NLTK's regex parser and chunker, a grammar such as
VP: {(<MD>|<R.*>|<I.*>|<VB.*>|<JJ.*>|
<TO>)*<VB.*>+(<MD>|<R.*>|<I.*>|<VB.*>|
<JJ.*>|<TO>)*}
with at least one verb, can extract relation-like phrases from the phrase that links two concepts. An example is shown in Figure FIGREF27. Further investigation of relation extraction from SE handbook is left as future work.
CONCLUSION AND FUTURE WORK
We presented a common-knowledge concept extractor for the Systems Engineer's Virtual Assistant (SEVA) system and showed how it can be beneficial for downstream tasks such as relation extraction and knowledge graph construction. We construct a word-level annotated dataset with the help of a domain expert by carefully defining a labelling scheme to train a sequence labelling task to recognize SE concepts. Further, we also construct some essential datasets from the SE domain which can be used for future research. Future directions include constructing a comprehensive common-knowledge relation extractor from SE handbook and incorporating such human knowledge into a more comprehensive machine-processable commonsense knowledge base for the SE domain.

Question: How many domain experts were involved into creation of dataset?
Answer: Unanswerable

1

One domain expert.

Question: What metrics are used for evaluation?
Answer: F1-score

precision, recall, f1-score, and support

Precision, recall, f1-score, and support.

Question: What is the performance of fine tuned model on this dataset?
Answer: F1-score of $0.89$

The model gives an F1-score of $0.89$ for the concept recognition task.

 F1-score of $0.89$

Question: Are constructed datasets open sourced?
Answer: Yes

Yes

Yes

Question: How does labeling scheme look like?
Answer: Based on operation and system concepts, the labels are abb, grp, syscon, opcon, seterm, event, org, art, cardinal, loc and mea.

BIO Labelling Scheme
abb: represents abbreviations such as TRL representing Technology Readiness Level.

grp: represents a group of people or an individual such as Electrical Engineers, Systems Engineers or a Project Manager.

syscon: represents any system concepts such as engineering unit, product, hardware, software, etc. They mostly represent physical concepts.

opcon: represents operational concepts such as decision analysis process, technology maturity assessment, system requirements review, etc.

seterm: represents generic terms that are frequently used in SE text and those that do not fall under syscon or opcon such as project, mission, key performance parameter, audit etc.

event: represents event-like information in SE text such as Pre-Phase A, Phase A, Phase B, etc.

org: represents an organization such as `NASA', `aerospace industry', etc.

art: represents names of artifacts or instruments such as `AS1300'

cardinal: represents numerical values such as `1', `100', 'one' etc.

loc: represents location-like entities such as component facilities or centralized facility.

mea: represents measures, features, or behaviors such as cost, risk, or feasibility.

1. abb
2. grp
3. syscon
4. opcon
5. seterm
6. event
7. org
8. art
9. cardinal
10. loc
11. mea

Question: What pretrained language model is used?
Answer: BERT

BERT

BERT

Question: How big is constructed dataset?
Answer: 3700 sentences

3700 sentences 

roughly 3700 sentences at the word-token level