Large Language Model(LLM) POC: Knowledge
Base Q&A
Databricks Professional Services

Summary
Large language models (LLMs) are the backbone of many natural language
processing (NLP) applications, such as ChatGPT and Google Translate. Most outof-the-box LLMs are general-purpose models trained on publicly available text.
However, many business problems need a specialized language model, augmented

Key benefits

and/or trained on domain-specific data sets, to deliver business value. ImagineI ncrease developer productivity

answering questions based on your organization’s knowledge base, integrated withR educe time to market

your Databricks Lakehouse.
Databricks can help you by:
Advising on how to build your own data set to get the most out of aI mprove security from keeping
your data and model in-houseR educe total cost of ownership(TCO)

large language model, such as LLama 2 or MPT
Building a knowledge base Q&A model prototype:C leaning data and creating a vector database for retrievalIdentifying and leveraging the best-suited open source LLMs
F ine-tuning a model with your data for custom embeddings(if necessary)B asic model evaluationA ny in-depth model evaluation requires manual inspection ofresults and collaboration with domain expertsE nabling internal teams
Setting up MLflow AI Gateway and understanding next steps to deploy
your Q&A model

Out of scopeC onfiguration and integration of
non-Databricks productsD ata cleansing associated withbuilding broader data lakeA ny efforts to improve an
existing benchmark model
have no guarantee that thenew model will outperform,
especially if the benchmark
model currently uses OpenAIA
 ny necessary PII removal
requires additional
implementation timeI ntegration of the Q&A bot withfront-end application

Key Outcome
A data strategy and reference implementation for a knowledge base Q&A
model jointly determined by customer and DatabricksI mplementing CI/CD andautomated deployment
infrastructure setup

Convert documents to embeddings

12

Load to Vector DB

ETL to Delta Table
0. 1
0.I
1.D
0.K
0.J
1.2C
 1...

Raw documents

Clean documents

Embedding ode

0. 10.I1.D
0.K
0.J
1.2C
 1...

SYlit documents“How to growblueberries?”

Engineer prompt with extended context
Knowing this context:-<Similar doc 1 from vector db-<Similar doc 2 from vector db Answer the question:
How to grow blueberries?

Index embeddings
Done once
Vector DB
embedding database
Similarity search
Add similar docs as context to the prompt

Summari¥ation ode

34

Create and log LangCfain cfain

Prerequisites
Data must exist in PDFs or in a tabular format, ideally withthese columns: title, content and web links (unless mutuallyagreed). Web scraping is out of scope. Additional text
cleaning for other file formats requires significantly more
implementation time.
Data for knowledge base is accessible by all company
employees in order to remove the risk of sensitive“Blueberries are most successfulwhen growing in acidic soil.
They like a lot of water and sun.”

Challenges
Building the right data set to fine-tune modelVector database integration (if not using
Databricks Vector Search)Evaluating models, plus alleviating safety and bias
concerns, is an active research community effort. Thereare currently no fully developed best practice standards.

information leakage via the Q&A bot
Availability of domain SMEs to collaborate with during theproject span, especially for model evaluation

Resources and schedule
HANDS-ON MVP
Up to 16 data scientist days spread over 4 weeks, with 1 day of project management time

Evaluate Databricks for yourself. Visit us at databricks.com and try Databricks for free!
© Databricks 2023. All rights reserved. Apache, Apache Spark, Spark and the Spark logo are trademarks of the Apache Software Foundation. Privacy Policy | Terms of Use