Extracting Information from 

Unstructured Data

Neil Alexander Kattampallil

Biocomplexity Institute, University of Virginia

Gary Anderson, National Center for Science and Engineering Statistics (NCSES), National 

Science Foundation
Unstructured Text Data : An Overview

Any information that doesn’t follow conventional data models, making it 

difficult to store and manage in a Relational Database.

• Word Documents

• Emails

• Social Media posts

• Medical notes (Healthcare NLP)

Most of this exists as ‘dark data’, data that is collected by an organization 

which is not used for analytics or insight generation, and is usually stored 

only for compliance purposes.
Unstructured Data as an information asset

"...a recent initiative to reduce unnecessary C-sections at a large health 

system in the Northwest. The first task for the team was to understand 

how the indications for C-section were documented in the Electronic 

Medical Record. It turned out that there were only two options to 

choose from: 1) fetal indication and 2) maternal indication. Because 

these were the only two options, delivering clinicians would often 

choose to document the true indication for C-section in a free text 

form, while others did not document it at all. "

Taken from Healthcatalyst.com, 5 Reasons Healthcare Data is Difficult to Measure
A pre-trained unsupervised Natural Language Processing model 

developed by Google in 2018

Methods: BERT

Image Attribution: Bert; Sesame Workshop
BERT Benefit

Pre-trained models solve challenge of lack of 

enough training data

Can use pre-trained 

models to fine tune on 

smaller-task specific 

datasets and improves 

accuracy drastically

Book Corpus:

2,500 million words Wikipedia:

800 million words

BERT Explained: A Complete Guide with Theory and Tutorials
How is BERT Trained?

Image Attribution: "What Have Language Models Learned" (huggingface.com)
Semantic Similarity

Language models like BERT are based on training with large data sets, such 

as:

• full text of Wikipedia

• Open Books corpus

The goal is to develop a vector representation of words, in a given sentence 

that represent a given idea.

These Language models contain vector representations of words, clustered 

by similarity of ideas in an n-dimensional vector space.
Goals for this presentation:

• The problem: Extracting information from text articles

• Why it is a problem: Lot of information is stored in the form of human 

readable text articles, which may not be easily accessible to automated 

systems, but this information is vital for research and analysis tasks.

• What has been done about the problem:

Currently, to measure Innovation, NCSES uses Surveys to businesses to 

obtain information about what their new products are.

• What you are doing (or have done) about the problem:

We are using news articles and press releases as a dataset, as companies 

often try to announce new product launches, and we can extract product 

name, company name and product features from articles.
Current Measure of Innovation

EuroStat: Community Innovation Survey

• Conducted every 2 years by EU 

member states and ESS member 

countries

• Survey of innovation activity by 

enterprises 

Annual Business Survey (US Census 

Bureau and NSF/NCSES)

• Collects data on R&D, innovation, 

technology, intellectual property, and 

business owner characteristics

• Annual; initial year: 2018

• Previously, Business R&D Innovation 

Survey (BRDIS) launched in 2009

Annual Business Survey (ABS) question on innovation
Background and Goals

• Can we use alternative (non-survey) data sources to measure business innovation?

– While ABS measures innovation incidence, i.e., the number of 

innovating firms, we aim to test the feasibility of developing methods using 

non-traditional data to obtain richer and complementary innovation measures.

• We develop natural language processing and machine learning methods using non-survey data to 

obtain richer and complementary innovation measures.

• Focus on opportunity and administrative data (e.g., product announcements, press releases, financial 

filings).

• What type(s) of innovation or improvement (i.e., greater efficacy, resource efficiency, reliability and 

resilience, affordability, convenience and usability) could be captured using opportunity data sources?

• How does it vary across different companies and sectors?
Current Focus

• Innovation Type: Product innovation as defined by the OSLO 

Manual

• Sectors: 

– Pharmaceutical and Medicine Manufacturing (NAICS 3254)

– Food Processing & Beverage Manufacturing (NAICS 311 & 3121)

– Computer Systems Design and Related Services (NAICS 5415)

• Data Sources: News articles

• Innovation Metrics: Number of new products/launches 

by company & measures of “novelty of innovation”

• Validation: Comparison of extracted metrics to ’ground-truth’ data 

and computation of performance metrics to 

evaluate methods

• Repeatability: Applying these data and methods to other sectors?

“ A product innovation is a new or improved 

good or service that differs significantly 

from the firm’s previous goods or services 

and that has been made available to 

potential users. “
• English language

• Articles between 2013 and 2021

• Region Code USA

• Variables: company_codes, subject_codes, 

region_codes, word_count

• Additional DNA Subject Codes that can be used to 

further improve data selection, e.g., c22 – New 

Product/Service

Data Sources

News articles from Dow Jones Data, News, 

and Analytics (DNA) database

Regulatory Agencies

Companies

Pharma Articles by Company (2015)

Pharma Articles by Publisher (2015)

Sector Number of articles

Pharmaceutical and Medicine Manufacturing 

(NAICS 3254)

1.8 M (2013 - 2018)

Food Processing & Beverage Manufacturing 

Sectors (NAICS 311 & 3121)

600 K (2013 - 2021)

Computer Systems Design and Related Services 

(NAICS 5415)

1.2 M (2013 - 2021)

Data Obtained
Named Entity Recognition

Named-entity recognition (NER), also known as

• (named) entity identification

• entity chunking, and entity extraction)

This is a subtask of information extraction that seeks to locate and classify named 

entities mentioned in unstructured text into pre-defined categories such as

• person names

• organizations, locations

• medical codes

• time expressions

• quantities

• monetary values

• percentages, etc.
Categories for NER

Normally, NER uses eight categories—location, person, organization, date, 

time, percentage, monetary value, and “none-of-the-above”. NER first finds 

named entities in sentences and declares the category of the entity. In the 

sentence:

“Apple [organization] CEO Tim Cook [Person] Introduces 2 New, Lager 

iPhones, Smart Watch at Cupertino [Location] Flint Center [Organization] 

Event.”

Note that “Apple” is recognized as an organization name instead of a fruit 

name in terms of its context.
BERT NER models

bert-base-NER is a fine-tuned BERT model that is ready to use for Named 

Entity Recognition and achieves state-of-the-art performance for the NER 

task. It has been trained to recognize four types of entities: location (LOC), 

organizations (ORG), person (PER) and Miscellaneous (MISC).

Specifically, this model is a bert-base-cased model that was fine-tuned on 

the English version of the standard CoNLL-2003 Named Entity Recognition 

dataset.
• Identifies and categorizes entities into 4 categories based on the 

context that the words are used in

– Location, Person, Organization, Miscellaneous

• Tokenizes words

– Craftsbury becomes Crafts## and ##bury

• Uses surrounding words (Sentence context)

– John works at _______.

• Is Case Sensitive

• Can be further fine-tuned by training on a corpus

BERT - Named Entity Recognition (NER)

Tagging by huggingface

model dlsim/bert-baseNER. 
• Use NER to extract organization names from the articles to identify company names

• Fuzzy Match (fuzzywuzzy) extracted names with DNA company names to evaluate NER

BERT - Named Entity Recognition (NER)
• Fuzzy Match (fuzzywuzzy) extracted names with DNA company names to evaluate NER

BERT - Named Entity Recognition (NER)
NER – Company Name Extraction 

• Performance Results for labeled data from 3 Sectors:

Pharma Food & 

Beverage Software

Total Number of Labeled Company Names: 284 129 232

Exact Matches: 110 36 112

Fuzzy Matches: 159 73 80

Total Matches: 269 109 192

Total Accuracy (%): 95% 84% 83%

Labeled data is a random sample of articles across all available years, as mentioned in slide 10
Training for Additional Categories

If there are custom categories that need to be extracted, we can use transfer 

learning to build on the existing categories that the model is trained for, and 

then add training data for new categories.
Applications of BERT:

Named Entity Recognition

21

https://towardsdatascience.com/named-entity-recognitionner-with-bert-in-spark-nlp-874df20d1d77
Bert - Question and Answering (QnA)

- BERT uses token embedding similarities to retrieve answers 

to a given question, from a given reference corpus of 

information.

• We use QnA to extract company and product names
BERT - Question and Answering (QnA)
Bert - Question and Answering (QnA)
QnA - Company and Product Extraction

Pharma Product Company

Total Number of Labeled Names: 284 284

Exact Matches: 140 88

Fuzzy Matches: 111 131

Total Matches: 251 219

Total Accuracy (%): 88.3% 77.1%
• Questions & Comments?

Contact:

Thomas Neil Alexander Kattampallil (nak3t@virginia.edu)

Research Scientist

Biocomplexity Institute, University of Virginia

Thank you!

Acknowledgments: This material is based on work supported by U.S. Department of Agriculture (58-3AEU-7-0074) 

and the National Science Foundation (Contract #49100420C0015)

Disclaimer: The views expressed in this paper are those of the authors and not necessarily those of their respective 

institutions.