NESTOR

MACHINE-AUGMENTED ANNOTATION FOR TECHNICAL TEXT

Thurston Sexton

Knowledge Extraction and Application for Smart Manufacturing Operations Management

Systems Integration Division

Engineering Laboratory
DISCLAIMER

The use of any products described in any presentation does not imply 

recommendation or endorsement by the National Institute of Standards 

and Technology, nor does it imply that products are necessarily the best 

available for the purpose.
PROJECT/PROGRAM OVERVIEW

Knowledge Extraction and Application

for Smart Manufacturing Operations Management

● Much of manufacturing know-how is computationally inaccessible, 

within informally-written documents

● Create human-centric data pipelines to extract value from existing 

unstructured data at minimal labor cost

● Develop guidelines for using semi-structured data in KPI creation, 

functional taxonomy prediction, and customized worker training 

paths 

3
MAINTENANCE WORK-ORDER DATA

“HP coolant pressure at

 75 psi”

“Major hydraulic leak at Sp#6 

horseshoe”

“Clamping spool guard 

broken”

“Hyd leak at saw 

atachment” 

“Replaced seal in saw 

attachment but still leaking 

– Reapirs pending with ML”

“Bad Gauge / Low pressure 

lines cleaned ou”

“Replaced – Operator could 

have done this!”

“Repaired horseshoe seals”

4
CURRENT MWO DATA ENTRY

5

Date Mach Description Issued By Date Up Maint Tech 

Assigned Resolution

29-Jan-16 H15 St#14 tool detect INOP JS 29-Nov-16 SA

Slug detector at station 14 not working. Would not 

recognize “Start” signal.

1-Jun-16 Mitsu 

FT

Brakes worn -Not 

stopping when in gear AB 28-Jun-16 Steve A Repaired

1-Jun-16 H8

St#7 rotator collet 

broken -wait for Bob B 

to show him how to 

remove

JS 8-Jun-16 John Smith

Machine went offline on 6/8 -Mark removed and 

instructed Bob B on removal/install process
Do “AI” to it!

But wait, we need to clean it first...
MANUAL CLEANING

7

“Major hydraulic 

leak at Sp#6 

horseshoe”

“Clamping 

spool guard 

broken”

“Hyd leak at saw 

atachment” 

“HP coolant 

pressure at

 75 psi”

“Replaced seal in saw 

attachment but still 

leaking – Reapirs 

pending with ML”

“Bad Gauge / 

Low pressure 

lines cleaned ou”

“Replaced – 

Operator could 

have done this!”

“Repaired 

horseshoe 

seals”

Description Resolution Cause Effect Solutions
MANUAL CLEANING

8

“Major hydraulic 

leak at Sp#6 

horseshoe”

“Clamping 

spool guard 

broken”

“Hyd leak at saw 

atachment” 

“HP coolant 

pressure at

 75 psi”

“Replaced seal in saw 

attachment but still 

leaking – Reapirs 

pending with ML”

“Bad Gauge / 

Low pressure 

lines cleaned ou”

“Replaced – 

Operator could 

have done this!”

“Repaired 

horseshoe 

seals”

Description Resolution Cause Effect Solutions

Hydraulic Leak at saw 

attachment

Replaced Seal; 

Repaired

Bad Gauge Low Coolant Cleaned Out Low 

Pressure Lines

Major Hydraulic Leak at 

SP#6 Horseshoe Repaired Horseshoe 

seals

Broken Clamping Spool 

Guard

Replaced Clamping 

Spool Guard

Operator error
MANUAL CLEANING

9

“Major hydraulic 

leak at Sp#6 

horseshoe”

“Clamping 

spool guard 

broken”

“Hyd leak at saw 

atachment” 

“HP coolant 

pressure at

 75 psi”

“Replaced seal in saw 

attachment but still 

leaking – Reapirs 

pending with ML”

“Bad Gauge / 

Low pressure 

lines cleaned ou”

“Replaced – 

Operator could 

have done this!”

“Repaired 

horseshoe 

seals”

Description Resolution Cause Effect Solutions

Hydraulic Leak at saw 

attachment

Replaced Seal; 

Repaired

Bad Gauge Low Coolant Cleaned Out Low 

Pressure Lines

Major Hydraulic Leak at 

SP#6 Horseshoe Repaired Horseshoe 

seals

Broken Clamping Spool 

Guard

Replaced Clamping 

Spool Guard

Operator error

12 hours for 

800 MWOs
PROBLEM

Enter Technical Language Processing

● NLP Techniques do not always adapt well to engineering text

● Current NLP solutions need to be adapted correctly for use in 

manufacturing

● TLP is a methodology to tailor NLP solutions to engineering text and 

industry use cases in a scalable and reproducible way [1]

10

[1] Brundage, Michael P., et al. "Technical language processing: Unlocking 

maintenance knowledge." Manufacturing Letters 27 (2020): 42-46.
TAGGING AS ANNOTATION

“Tags” have many benefits for users:

● No Controlled Vocabulary

● Naturally reflect the user-base’s communication

● Correct abstraction level → more usage

Collections of tags on a domain form a “Folksonomy”

● Relationships are encoded via tag co-occurrence — like Bag of Words!

● Can be predicted via Multi-label classification

● Compare many user’s annotations for customizable validation 

We sacrifice certainty about specific labels, and gain annotation quality & ease-of-use
BETTER, BUT WHAT NEXT?

12 hrs

800 annotated (?)

12 hrs

1200 annotated

Cause/Effects

By Hand

Prob./Item/Soltn.

Tagging
Ranked Concept Tagging

The Nestor Workflow
RANKED CONCEPT TAGGING

14

“Major hydraulic 

leak at Sp#6 

horseshoe”

“Clamping 

spool guard 

broken”

“Hyd leak at saw 

atachment” 

“HP coolant 

pressure at

 75 psi”

“Replaced seal in saw 

attachment but still 

leaking – Reapirs 

pending with ML”

“Bad Gauge / 

Low pressure 

lines cleaned ou”

“Replaced – 

Operator could 

have done this!”

“Repaired 

horseshoe 

seals”

Description Resolution
15

token type alias

repaired

replaced

leak

seal

hydraulic

bad

gauge

low pressure

reapirs

hyd

More 

Important

“Major hydraulic 

leak at Sp#6 

horseshoe”

“Clamping 

spool guard 

broken”

“Hyd leak at saw 

atachment” 

“HP coolant 

pressure at

 75 psi”

“Replaced seal in saw 

attachment but still 

leaking – Reapirs 

pending with ML”

“Bad Gauge / 

Low pressure 

lines cleaned ou”

“Replaced – 

Operator could 

have done this!”

“Repaired 

horseshoe 

seals”

Description Resolution

RANKED CONCEPT TAGGING
16

token type alias

repaired

replaced

leak

seal

hydraulic

bad

gauge

low pressure

reapirs

hyd

Similar 

Words

Same as 

Repaired? 

(Y/N)

Reapirs

Repair

Repir

Rep

Reparis

Repaird

RANKED CONCEPT TAGGING

“Major hydraulic 

leak at Sp#6 

horseshoe”

“Clamping 

spool guard 

broken”

“Hyd leak at saw 

atachment” 

“HP coolant 

pressure at

 75 psi”

“Replaced seal in saw 

attachment but still 

leaking – Reapirs 

pending with ML”

“Bad Gauge / 

Low pressure 

lines cleaned ou”

“Replaced – 

Operator could 

have done this!”

“Repaired 

horseshoe 

seals”

Description Resolution
17

token type alias

repaired

replaced

leak

seal

hydraulic

bad

gauge

low pressure

reapirs

hyd

Similar 

Words

Same as 

Repaired? 

(Y/N)

Reapirs Y

Repair Y

Repir Y

Rep N

Reparis Y

Repaird Y

RANKED CONCEPT TAGGING

“Major hydraulic 

leak at Sp#6 

horseshoe”

“Clamping 

spool guard 

broken”

“Hyd leak at saw 

atachment” 

“HP coolant 

pressure at

 75 psi”

“Replaced seal in saw 

attachment but still 

leaking – Reapirs 

pending with ML”

“Bad Gauge / 

Low pressure 

lines cleaned ou”

“Replaced – 

Operator could 

have done this!”

“Repaired 

horseshoe 

seals”

Description Resolution
18

token type alias

repaired S repaired

replaced

leak

seal

hydraulic

bad

gauge

low pressure

reapirs S repaired

hyd

Similar 

Words

Same as 

Repaired? 

(Y/N)

Reapirs Y

Repair Y

Repir Y

Rep N

Reparis Y

Repaird Y

RANKED CONCEPT TAGGING

“Major hydraulic 

leak at Sp#6 

horseshoe”

“Clamping 

spool guard 

broken”

“Hyd leak at saw 

atachment” 

“HP coolant 

pressure at

 75 psi”

“Replaced seal in saw 

attachment but still 

leaking – Reapirs 

pending with ML”

“Bad Gauge / 

Low pressure 

lines cleaned ou”

“Replaced – 

Operator could 

have done this!”

“Repaired 

horseshoe 

seals”

Description Resolution
token type alias

repairedS repaired

replacedS

leakP

seal I

hydraulic I

badP

gauge I

low pressureP

reapirs I repaired

hyd I

19

RANKED CONCEPT TAGGING

“Major hydraulic 

leak at Sp#6 

horseshoe”

“Clamping 

spool guard 

broken”

“Hyd leak at saw 

atachment” 

“HP coolant 

pressure at

 75 psi”

“Replaced seal in saw 

attachment but still 

leaking – Reapirs 

pending with ML”

“Bad Gauge / 

Low pressure 

lines cleaned ou”

“Replaced – 

Operator could 

have done this!”

“Repaired 

horseshoe 

seals”

Description Resolution
20

“Major hydraulic

leak at Sp#6

horseshoe”

“Clamping

spool guard

broken”

“Hyd leak at saw

atachment” 

“HP coolant

pressure at

 75 psi”

“Replaced seal in saw

attachment but still 

leaking – Reapirs

pending with ML”

“Bad Gauge / 

Low pressure

lines cleaned ou”

“Replaced –

Operator could 

have done this!”

“Repaired

horseshoe

seals”

Description Resolution

More 

Important

token type alias

repaired S repaired

replaced S

leak P

seal I

hydraulic I

bad P broken

gauge I

low pressure P

reapirs I repaired

hyd I hydraulic

RANKED TAGGING
21

“Major hydraulic

leak at Sp#6 

horseshoe”

“Clamping 

spool guard 

broken”

“Hyd leak at saw 

atachment” 

“HP coolant 

pressure at

 75 psi”

“Replaced seal in saw 

attachment but still 

leaking – Reapirs 

pending with ML”

“Bad Gauge / 

Low pressure 

lines cleaned ou”

“Replaced – 

Operator could 

have done this!”

“Repaired 

horseshoe 

seals”

Description Resolution Item(s) Problem(s) Solution(s)

Hydraulic

Hydraulic

RANKED TAGGING
22

“Major hydraulic

leak at Sp#6 

horseshoe”

“Clamping 

spool guard 

broken”

“Hyd leak at saw 

atachment” 

“HP coolant 

pressure at

 75 psi”

“Replaced seal in saw 

attachment but still 

leaking – Reapirs 

pending with ML”

“Bad Gauge / 

Low pressure 

lines cleaned ou”

“Replaced – 

Operator could 

have done this!”

“Repaired 

horseshoe 

seals”

Description Resolution Item(s) Problem(s) Solution(s)

Hydraulic

Hydraulic

Replaced

Replaced

RANKED TAGGING
23

Description Resolution Item(s) Problem(s) Solution(s)

Leak Replaced; 

Repaired

Broken; Low Pressure

High Pressure Coolant; 

Gauge; Low Pressure 

Line 

Cleaned

Leak Repaired

Broken Replaced Clamping Spool Guard; 

Operator

Hydraulic; Saw 

attachment; Seal

Hydraulic; SP#6 

Horseshoe, Seal 

“Major hydraulic

leak at Sp#6

horseshoe”

“Clamping

spool guard

broken”

“Hyd leak at saw

atachment” 

“HP coolant

pressure at

 75 psi”

“Replaced seal in saw

attachment but still 

leaking – Reapirs

pending with ML”

“Bad Gauge / 

Low pressure

lines cleaned ou”

“Replaced –

Operator could 

have done this!”

“Repaired

horseshoe

seals”

RANKED TAGGING
24

Description Resolution Item(s) Problem(s) Solution(s)

Leak Replaced; 

Repaired

Broken; Low Pressure

High Pressure Coolant; 

Gauge; Low Pressure 

Line 

Cleaned

Leak Repaired

Broken Replaced Clamping Spool Guard; 

Operator

Hydraulic; Saw 

attachment; Seal

Hydraulic; SP#6 

Horseshoe, Seal 

“Major hydraulic

leak at Sp#6

horseshoe”

“Clamping

spool guard

broken”

“Hyd leak at saw

atachment” 

“HP coolant

pressure at

 75 psi”

“Replaced seal in saw

attachment but still 

leaking – Reapirs

pending with ML”

“Bad Gauge / 

Low pressure

lines cleaned ou”

“Replaced –

Operator could 

have done this!”

“Repaired

horseshoe

seals”

RANKED TAGGING

0.75 hours for 

all MWOs
RANKED CONCEPT TAGGING

12 hrs

800 annotated (?)

12 hrs

1200 annotated

Cause/Effects

By Hand

Prob./Item/Soltn.

Tagging

Ranked Concept

Tagging

0.75 hrs

80% + (3k/3.5k)

99% partials
Nestor

reference implementation

Prototype GUI

https://github.com/usnistgov/nestor

Next: When to use? How to 

properly treat tag-style data? 🡪

Guidelines 

PUTTING IT TOGETHER
GENERALIZE: LESSONS FROM INFO-VIZ

Mantra: Visual Information Seeking

Overview First

Zoom & Filter

Details-on-Demand
Learn from Information Visualization!

Mantra: Visual Information Seeking

Overview First

Zoom & Filter

Details-on-Demand

Augmented Knowledge Extraction

Extract & Rank Concepts

Detect & Filter Relations

Context as Needed

- Unsupervised

- Fortuitous Data

- “Domain” or “Stats”

- Defeat the “Hairball”

- Original?

- Abstractions?
What Now?

How do I use “tags”
Exploratory Data Analysis

USING TAGS
Survival Analysis

● MTBF Estimation: 

On-par with rules-based 

systems! [2]

● Coming soon: 

Data quality benchmarking and 

Human-centric modeling

[Conte et al., under review]

USING TAGS

[2] Sexton, Thurston, et al. "Benchmarking for keyword extraction methodologies in 

maintenance work orders." PHM society conference. Vol. 10. 2018.
HVAC: Time Series and 

Impact Analyses

USING TAGS

● Quantify intervention impacts with 

suitable tag combinations

● “Hackable”: easy to combine with various 

visualizations and query techniques.

QUESTIONS?

Thurston Sexton

thurston.sexton@nist.gov
Copy and paste icon to desired slide. To change color, double click on icon, select color from drop down. For 

consistency, please use colors in the template. Due to licensing restrictions, you can only use these icons for 

NIST PowerPoints.