NLP Use Case Database Project
A bottom-up qualitative NLP success/failure database

Etzard Stolte, PhD, Roche
Bikalpa Neupane, PhD, Takeda
Birthe Nielsen, PhD, Pistoia Alliance
NLPdatabase@pistoiaalliance.org

Agenda• The team• The problem• The project• What has been achieved to date

• What is planned• Get involved

2© Pistoia Alliance

Collaborate toInnovate
Conference
Boston 2022

NLP Use Case Database

Project manager: Birthe Nielsen
nlpdatabase@pistoiaalliance.org

Problem Statement:

Project champion:

Pharma companies apply NLP methods in hopes of automation and insightgeneration. Although NLP algorithms have matured quite a bit during thepast years, practical value for most NLP pilots tends to be poor, and very few
NLP-driven projects are seen through to production. Exceptions are typically
topics w/ good metadata quality, large training sets and willing businesscolleagues to verify results, and a serendipitous combination of technicalexpertise and suitable use cases.

Etzard Stolte, Roche ; Bikalpa Neupane, Takeda.

Steering committee (Currently forming):

Value Proposition:
This type of knowledge is of value to share in a pre-competitive manner
among Pistoia Alliance members. A simple database could containcharacterization of use case, data characterization, pipelines & algorithms
used, quality criteria, outcomes, comments, etc. The value for participating
members would be a reference database to help narrow down successful
use case scenarios, less experimentation, and more successes.
Project member roles: Global Knowledge Management, Director ofAdvanced Technologies, Head of Text Analytics, Literature-based ScientificDiscovery, Global Information and Analysis, NLP Lead/Engineer, Data
Scientist
Q1 2022
Q2 2022
Q4 2022
Q3 2022
Roundtable Discussion

Presentation of UseCases

Draft Database Created, Populate 50-100 Use Cases
First Use Cases Added$5-15K ensures a seat on the Steering Committee

Project deliverables:•••

A bottom-up qualitative NLP Success Failure Database
Agreed annotations of NLP use case methods andsuccess/failure criteria
Collaborative insight into why NLP use cases may fail orsucceed with an industry –wide view
2023
Standardize nomenclature, continue populating, analyze successprediction criteriahttps://drive.google.com/file/d/1AkmetDVgCH92zBJ4HZjuVhSgn5H1Xq
ZU/view?usp=sharing

Roche ideas
proposalRound tablediscussion withinterested
pharmamembers

Draft Use Case
Database

Q1 2022

Q3 2022First Use cases
addedWorking groupmeeting toagree on
annotation of
NLP use casemethods andsuccess/failure
criteria

Collection ofuse cases to becompleted (50100 Use cases)

Collaborative
insight to why
NLP use cases
may fail ofsucceed – wider
view.
Next steps
discussion

Q3 2022

Q4 2022

Q1 20235© Pistoia Alliance

Project timeline till date

Machine Learning (NLP, NER) uses cases in Manufacturing© Pistoia Alliance

Wide range of use cases

Hype Cycle for AI, ML, NLP
Gartner, 2020: real world applications of AI/ML/NLP struggle with quality© Pistoia Alliance

Quality!

Natural Language Processing (NLP)
Challenges ? Hell yes - it simply does not deliver the quality we need in Pharma• Pharma is moving beyond buzzwords tovalue generation• NLP is more of an art, then engineering• Observations– Hard to determine up front, if a use-case willwork– “successful” pipelines do not work for slightly
different data

– Manual curation, training sets are exceedingly
work-intensive

the quality we need

© Pistoia Alliance– Success seems very use case specific– Expensive to find talent that actually delivers

Pistoia Alliance - NLP Expert Group
Value Proposition• Share best-practice – get an idea what works &what does not work
• Build database of use cases to be shared in thisgroup; incl. failures• Obvious challenges to address– Can we come up with a categorization thatmakes sense?
– If NLP is an art, can we structure use-cases in ameaningful way?

– Can we deliver on the effort to identify andenter your use cases iteratively into somedatabase?

Systema naturæ sistens regna tria naturæ, in classes et ordines, genera et species
redacta tabulisque æneis illustrate, Systema Naturae, 1768, Carl von Linné© Pistoia Alliance– Is a phenotypic classification sufficient (aka likeLinne)?

Source: https://landscape.lfai.foundation/card-mode?category=natural-language-processing&grouping=category© Pistoia Alliance

Natural Language Processing LandscapeMeaning Based ComputingConcepts

Trend Detection

Quality Controls

Risk Prediction

Cause Analysis

Automatically extract
and correlate concepts
using semantic tools

Interactively explore
emerging concepts /trends

Monitor operational &content quality measureshidden in un-structured
information

Real-time classifiers tosense risks across sources,
sites & organizations

Extract relationships
among concepts in a
Knowledge Graph• find documents/experts• Automatically add quality
tags to lab analysis• Show potentially recurring
deviations for new
TrackWise entries• sense patterns in healthauthority records• early warning signs forbatch or release records• off-label use of products• detect mis-filed
documents in submission
structures• flag spurious
relationships inmanufacturing logs

• sentiment analysis of
HA ePARs with respect
to submission success• emerging issues inbatch records• understand cause /effect relationships inmanufacturing
deviations• visualise relationships /dependencies insideSOPs

11© Pistoia Alliance

Examples

TypeUse Cases types and examples

1

The Use Case Database12© Pistoia AllianceContributions till date:Where our current database stands today▪ Named-entity recognition (NER)▪ Concept Extraction (Taxonomy extraction,© Pistoia Alliance▪
▪
▪
▪
▪

concept tagging, document classification)
Text-Classification
Unsupervised topic modeling
Extractive auto-summarization
Similarity Search
Chatbots/Virtual Assistants

•

•

•

Use case– Deployment type
– Project level– Company– Language– etc
Data– Description– Size– Issues– Preparation– FTE involved/team– etc
Algorithm– Summary– Version– Library– etc

Database content•••

Outcome– Learnings– Success achieved– etcDiscussions– Effort/ benefit ratio– Non-NLP approach– Success– SupportOutlook– Build vs buy recommendation– Other applications of technology– etc14© Pistoia Alliance1

Pistoia Alliance - NLP Expert Group
What do we NOT want to do• Encyclopedia of NLP methods• Another NLP discussion group• Vendor messages, vision documents,
general statements• Evangelism of a particular NLP schoolof thought or technology• ???

© Pistoia Alliancehttps://mattturck.com/, November 2021 – Machine Learning & Data LandscapeNLP Build vs Buy❑ Language is hard – ambiguous, Challenges because human expression is diverse, often

ambiguous, affected by age, demographics, socio-economics, medium/channel & regional
attributes of the author.

❑ Building a POC NLP model is trivial. However, gathering domain specific dataset, training
large models and deploying them reliability (serving thousands of requests at low latency)
is extremely hard.

❑ Running Experiments in Parallel and Serving Models is computationally expensive❑ Market Barriers – Lack of Clean and accurate data - Lack of human-experts;❑ Compliance is not trivial (PII, PHI, GDPR, FERPAA) – Customizability, AI Explainability, andFairness are daunting challenges to fulfill without a large, in-house data science team© Pistoia Alliance❑ Head Count + Maintenance Costs + Infrastructure Costs ➔ ROI ?

What if you were to purchase, checklist© Pistoia Alliance❑ Return on Investment (ROI)❑ Domain Fit Accuracy of the results❑ Production Readiness - Scalability and Performance, Deployment Options❑ Interoperability and Fit with existing software stack❑ Integration with existing ML pipeline❑ Flexibility - NLP custom models❑ Usability❑ Industry Leaders – Look at who is doing the real NLP work
❑ Compare and contrast regular use cases

We are eager to have you….

More information here:•

https://www.pistoiaalliance.org/projects/current-projects/natural-languageprocessing-use-case-database/Get in touch:18© Pistoia Alliance• nlpdatabase@pistoiaalliance.org

Questions

19© Pistoia Alliancenlpdatabase@pistoiaalliance.org