Package: blendsql
SQL 🤝 LLMsCheck out ouronline documentationfor a more comprehensive overview.IntroBlendSQL is asuperset of SQLitefor problem decomposition and hybrid question-answering with LLMs. It builds off of the syntax of SQL to create an intermediate representation for tasks requiring complex reasoning over both structured and unstructured data.It can be viewed as an inversion of the typical text-to-SQL paradigm, where a user calls a LLM, and the LLM calls a SQL program.
Here, the user is given the control to oversee all calls (LLM + SQL) within a unified query language.For example, imagine we have the following tables.wdaterivalcityvenuescore31 maynsw waratahssydneyagricultural society ground11-05 junnorthern districtsnewcastlesports ground29-07 junnsw waratahssydneyagricultural society ground21-211 junwestern districtsbathurstbathurst ground11-012 junwallaroo & university nswsydneycricket ground23-10documentstitlecontentsydneysydney ( /ˈsɪdni/ ( listen ) sid-nee ) is the ...new south wales waratahsthe new south wales waratahs ( /ˈwɒrətɑːz/ or ...sydney showground (moore park)the former sydney showground ( moore park ) at...sydney cricket groundthe sydney cricket ground ( scg ) is a sports ...newcastle, new south walesthe newcastle ( /ˈnuːkɑːsəl/ new-kah-səl ) met...bathurst, new south walesbathurst /ˈbæθərst/ is a city in the central t...BlendSQL allows us to ask the following questions by injecting "ingredients", which are callable functions denoted by double curly brackets ({{,}}).
The below examples work out of the box, but you are able to design your own ingredients as well!What was the result of the game played 120 miles west of Sydney?SELECT*FROMwWHEREcity={{LLMQA('Which city is located 120 miles west of Sydney?',(SELECT*FROMdocumentsWHEREdocumentsMATCH'sydney OR 120'),options='w::city')}}Which venues in Sydney saw more than 30 points scored?SELECTDISTINCTvenueFROMwWHEREcity='sydney'AND{{LLMMap('More than 30 total points?','w::score')}}=TRUEShow all NSW Waratahs games and a description of the team.SELECTdate,rival,score,documents.contentAS"Team Description"FROMwJOIN{{LLMJoin(left_on='documents::title',right_on='w::rival')}}WHERErival='nsw waratahs'More Examples from Popular QA DatasetsHybridQAFor this setting, our database contains 2 tables: a table from Wikipediaw, and a collection of unstructured Wikipedia articles in the tabledocuments.What is the state flower of the smallest state by area ?SELECT"common name"AS'State Flower'FROMwWHEREstate={{LLMQA('Which is the smallest state by area?',(SELECTtitle,contentFROMdocuments),options='w::state')}}Who were the builders of the mosque in Herat with fire temples ?{{LLMQA('Name of the builders?',(SELECTtitleAS'Building',contentFROMdocumentsWHEREtitle={{LLMQA('Align the name to the correct title.',(SELECTnameFROMwWHEREcity='herat'ANDremarksLIKE'%fire temple%'),options='documents::title')}}))}}What is the capacity of the venue that was named in honor of Juan Antonio Samaranch in 2010 after his death ?SELECTcapacityFROMwWHEREvenue={{LLMQA('Which venue is named in honor of Juan Antonio Samaranch?',(SELECTtitleAS'Venue',contentFROMdocuments),options='w::venue')}}OTT-QAUnlike HybridQA, these questions are open-domain, where we don't know in advance where the answer of a given open question appears in a passage or a table.As a result, we need to play the role of both the retriever (to select relevant context) and reader (to read from relevant contexts and return the given answer).As the underlying database consists of 400K tables and 5M documents, it's important to setLIMITclauses appropriately to ensure reasonable execution times.The examples below also demonstrate how BlendSQL unpacksCTE statementsto ensure we only pass necessary data into the BlendSQL ingredient calls.When was the third highest paid Rangers F.C . player born ?{{LLMQA('When was the Rangers Player born?'(WITHtAS(SELECTplayerFROM(SELECT*FROM"./List of Rangers F.C. records and statistics (0)"UNIONALLSELECT*FROM"./List of Rangers F.C. records and statistics (1)")ORDERBYtrim(fee,'£')DESCLIMIT1OFFSET2),dAS(SELECT*FROMdocumentsJOINtWHEREdocumentsMATCHt.player||' OR rangers OR fc'ORDERBYrankLIMIT5)SELECTd.content,t.playerAS'Rangers Player'FROMdJOINt))}}In which Track Cycling World Championships event was the person born in Matanzas , Cuba ranked highest ?{{LLMQA('In what event was the cyclist ranked highest?',(SELECT*FROM(SELECT*FROM"./Cuba at the UCI Track Cycling World Championships (2)")aswWHEREw.name={{LLMQA("Which cyclist was born in Matanzas, Cuba?",(SELECT*FROMdocumentsWHEREdocumentsMATCH'matanzas AND (cycling OR track OR born)'ORDERBYrankLIMIT3),options="w::name")}}),options='w::event')}}Who is the director the Togolese film that was a 30 minute film that was shot in 16mm ?SELECTdirectorFROM"./List of African films (4)"aswWHEREtitle={{LLMQA('What is the name of the Togolese film that was 30 minutes and shot in 16mm?'(SELECT*FROMdocumentsWHEREdocumentsMATCH'togolese OR 30 OR 16mm OR film'ORDERBYrankLIMIT5)options='w::title')}}FEVEROUSHere, we deal not with questions, but truth claims given a context of unstructured and structured data.These claims should be judged as "SUPPORTS" or "REFUTES". Using BlendSQL, we can formulate this determination of truth as a function over facts.Oyedaea is part of the family Asteraceae in the order Asterales.SELECTEXISTS(SELECT*FROMw0WHERE"family:"='asteraceae'AND"order:"='asterales')The 2006-07 San Jose Sharks season, the 14th season of operation (13th season of play) for the National Hockey League (NHL) franchise, scored the most points in the Pacific Division.SELECT({{LLMValidate('Is the Sharks 2006-07 season the 14th season (13th season of play)?',(SELECT*FROMdocuments))}})AND(SELECT(SELECTfilledcolumnnameFROMw0ORDERBYptsDESCLIMIT1)='san jose sharks')Saunders College of Business, which is accredited by the Association to Advance Collegiate Schools of Business International, is one of the colleges of Rochester Institute of Technology established in 1910 and is currently under the supervision of Dean Jacqueline R. Mozrall.SELECTEXISTS(SELECT*FROMw0WHERE"parent institution"='rochester institute of technology'AND"established"='1910'AND"dean"='jacqueline r. mozrall')AND({{LLMValidate('Is Saunders College of Business (SCB) accredited by the Association to Advance Collegiate Schools of Business International (AACSB)?',(SELECT*FROMdocuments))}})Table of ContentsInstallQuickstartFAQDocumentationExecute a BlendSQL QuerySmoothieIngredientsMapIngredientQAIngredientConstrained Decoding with 'options'JoinIngredientStringIngredientLLMsDatabasesAppendixFeaturesSmart parsing optimizes what is passed to external functions 🧠Traverses AST withsqlglotto minimize external function callsAccelerated LLM calls, caching, and constrained decoding 🚀Enabled viaguidanceEasy logging of execution environment withsmoothie.save_recipe()🖥️Enables reproducibility across machinesFor a technical walkthrough of how a BlendSQL query is executed, check outtechnical_walkthrough.md.Installpip install blendsqlQuickstartfromblendsqlimportblend,LLMQA,LLMMapfromblendsql.dbimportSQLiteDBConnectorfromblendsql.llmsimportOpenaiLLMblendsql="""SELECT merchant FROM transactions WHERE{{LLMMap('is this a pizza shop?', 'transactions::merchant'}} = TRUEAND parent_category = 'Food'"""# Make our smoothie - the executed BlendSQL scriptsmoothie=blend(query=blendsql,blender=OpenaiLLM("gpt-3.5-turbo-0613"),ingredients={LLMMap,LLMQA},db=SQLiteDBConnector(db_path="transactions.db"),verbose=True)FAQWhy not just implement BlendSQL as auser-defined function in SQLite?LLMs are expensive, both in terms of $ cost and compute time. When applying them to SQLite databases, we want to take special care in ensuring we're not applying them to contexts where they're not required.
This isnot easily achievable with UDFs, even when marked as adeterministic function.BlendSQL is specifically designed to enforce an order-of-operations that 1) prioritizes vanilla SQL operations first, and 2) caches results from LLM ingredients so they don't need to be recomputed.
For example:SELECT{{LLMMap('What state is this NBA team from?','w::team')}FROMwWHEREnum_championships>3ORDERBY{{LLMMap('What state is this NBA team from?','w::team')}BlendSQL makes sure to only pass thoseteamvalues from rows which satisfy the conditionnum_championship > 3to the LLM. Additionally, since we assume the function is deterministic, we make a single LLM call and cache the results, despite the ingredient function being used twice.So I get how to write BlendSQL queries. But why would I use this over vanilla SQLite?Certain ingredients, likeLLMJoin, will likely give seasoned SQL experts a headache at first. However, BlendSQL's real strength comes from it's use as anintermediate representation for reasoning over structured + unstructured with LLMs. Some examples of this can be found abovehere.Documentation[!WARNING]
WIP, will be updatedExecute a BlendSQL QueryTheblend()function is used to execute a BlendSQL query against a database and return the final result, in addition to the intermediate reasoning steps taken.::: blendsql.blendsql.blend
handler: pythonfromblendsqlimportblend,LLMMap,LLMQA,LLMJoinfromblendsql.dbimportSQLiteDBConnectorfromblendsql.llmsimportOpenaiLLMblendsql="""SELECT * FROM wWHERE city = {{LLMQA('Which city is located 120 miles west of Sydney?',(SELECT * FROM documents WHERE documents MATCH 'sydney OR 120'),options='w::city')}}"""db=SQLiteDBConnector(db_path)smoothie=blend(query=blendsql,db=db,ingredients={LLMMap,LLMQA,LLMJoin},blender=AzureOpenaiLLM("gpt-4"),# Optional args belowinfer_map_constraints=True,silence_db_exec_errors=False,verbose=True,blender_args={"few_shot":True,"temperature":0.01})SmoothieThesmoothie.pyobject defines the output of an executed BlendSQL script.@dataclassclassSmoothie:df:pd.DataFramemeta:SmoothieMeta@dataclassclassSmoothieMeta:process_time_seconds:floatnum_values_passed:int# Number of values passed to a Map/Join/QA ingredientnum_prompt_tokens:int# Number of prompt tokens (counting user and assistant, i.e. input/output)prompts:List[str]# Log of prompts submitted to modelexample_map_outputs:List[Any]# outputs from a Map ingredient, for debuggingingredients:List[Ingredient]query:strdb_path:strcontains_ingredient:bool=Truedefblend(*args,**kwargs)->Smoothie:...IngredientsIngredients are at the core of a BlendSQL script.They are callable functions that perform one the task paradigms defined iningredient.py.At their core, these are not a new concept.User-defined functions (UDFs), orApplication-Defined Functions in SQLitehave existed for quite some time.However, ingredients in BlendSQL are intended to be optimized towards LLM-based functions, defining an order of operations for traversing the AST such that the minimal amount of data is passed into your expensive GPT-4/Llama 2/Mistral 7b/etc. prompt.Ingredient calls are denoted by wrapping them in double curly brackets,{{ingredient}}.The following ingredient types are valid.MapIngredientThis type of ingredient applies a function on a given table/column pair to create a new column containing the function output.For example, take the following query.SELECTmerchantFROMtransactionsWHERE{{LLMMap('Is this a pizza shop?','transactions::merchant')}}=TRUELLMMapis one of our builtin MapIngredients. For each of the distinct values in the "merchant" column of the "transactions" table, it will create a column containing the function output.merchantIs this a pizza shop?Domino's1Safeway0Target0The temporary table shown above is then combined with the original "transactions" table with anINNER JOINon the "merchant" column.JoinIngredientHandles the logic of semanticJOINclauses between tables.For example:SELECTCapitals.name,State.nameFROMCapitalsJOIN{{LLMJoin('Align state to capital',left_on='States::name',right_on='Capitals::name')}}The above example hints at a database schema that would makeE.F Coddvery angry: why do we have two separate tablesStatesandCapitalswith no foreign key to join the two?BlendSQL was built to interact with tables "in-the-wild", and many (such as those on Wikipedia) do not have these convenient properties of well-designed relational models.For this reason, we can leverage the internal knowledge of a pre-trained LLM to do theJOINoperation for us.QAIngredientSometimes, simply selecting data from a given database is not enough to sufficiently answer a user's question.TheQAIngredientis designed to return data of variable types, and is best used in cases when we either need:Unstructured, free-text responses ("Give me a summary of all my spending in coffe")Complex, unintuitive relationships extracted from table subsets ("How many consecutive days did I spend in coffee?")The following query demonstrates usage of the builtinLLMQAingredient.{{LLMQA('How many consecutive days did I buy stocks in Financials?',(SELECTaccount_history."Run Date",account_history.Symbol,constituents."Sector"FROMaccount_historyLEFTJOINconstituentsONaccount_history.Symbol=constituents.SymbolWHERESector="Financials"ORDERBY"Run Date"LIMIT5))}}This is slightly more complicated than the rest of the ingredients.Behind the scenes, we wrap the call toLLMQAin a trivialCASEclause, ensuring that the ingredient's output gets returned.SELECTCASEWHENFALSETHENFALSEWHENTRUEthen{{QAIngredient}}ENDThe LLM gets both the question asked, alongside the subset of the SQL database fetched by our subquery."Run Date"SymbolSector2022-01-14HBANFinancials2022-01-20AIGFinancials2022-01-24AIGFinancials2022-01-24NTRSFinancials2022-01-25HBANFinancialsFrom examining this table, we see that we bought stocks in the Financials sector 2 consecutive days (2022-01-24, and 2022-01-25).
The LLM answers the question in an end-to-end manner, returning the result2.TheQAIngredientcan be used as a standalone end-to-end QA tool, or as a component within a larger BlendSQL query.For example, the BlendSQL query below translates to the valid (but rather confusing) question:"Show me stocks in my portfolio, whose price is greater than the number of consecutive days I bought Financial stocks multiplied by 10. Only display those companies which offer a media streaming service."SELECTSymbol,"Last Price"FROMportfolioWHERE"Last Price">{{LLMQA('How many consecutive days did I buy stocks in Financials?',(SELECTaccount_history."Run Date",account_history.Symbol,constituents."Sector"FROMaccount_historyLEFTJOINconstituentsONaccount_history.Symbol=constituents.SymbolWHERESector="Financials"ORDERBY"Run Date"LIMIT5))}}*10AND{{LLMMap('Offers a media streaming service?','portfolio::Description')}}=1Constrained Decoding withoptionsPerhaps we want the answer to the above question in a different format. We call our LLM ingredient in a constrained setting by passing aoptionsargument, where we provide either semicolon-separated options, or a reference to a column.{{LLMQA('How many consecutive days did I buy stocks in Financials?',(SELECTaccount_history."Run Date",account_history.Symbol,constituents."Sector"FROMaccount_historyLEFTJOINconstituentsONaccount_history.Symbol=constituents.SymbolWHERESector="Financials"ORDERBY"Run Date"LIMIT5)options='one consecutive day!;two consecutive days!;three consecutive days!')}}Running the above BlendSQL query, we get the outputtwo consecutive days!.Thisoptionsargument can also be a reference to a given column.For example (from theHybridQA dataset):SELECTcapacityFROMwWHEREvenue={{LLMQA('Which venue is named in honor of Juan Antonio Samaranch?',(SELECTtitle,contentFROMdocumentsWHEREcontentLIKE'%venue%'),options='w::venue')}}Or, from our running example:{{LLMQA('Which did i buy the most?',(SELECTaccount_history."Run Date",account_history.Symbol,constituents."Sector"FROMaccount_historyLEFTJOINconstituentsONaccount_history.Symbol=constituents.SymbolWHERESector="Financials"ORDERBY"Run Date"LIMIT5)options='account_history::Symbol')}}The above BlendSQL will yield the resultAIG, since it appears in theSymbolcolumn fromaccount_history.StringIngredientThis is the simplest type of ingredient. This will output a string to be placed directly into the SQL query.We have theDTfunction as a builtin StringIngredient.SELECTmerchantFROMtransactionsWHERE{{DT('transactions::date',start='q2')}}This will call a Python function that usesdatetimeto interpret the absolute dates which the relative phrase "q2" most likely refers to.We do not create any new tables or perform any joins with a StringIngredient; instead, we simply get the following SQL query.[!NOTE]
The below SQL interpretation of theDTfunction assumes we're calling it in December, 2022. The phrase 'q2' will be interpreted differently in, say, March 1998.SELECTmerchantFROMtransactionsWHEREdate>'2022-09-30'ANDdate<'2022-12-01'LLMsDatabases### Appendix
#### Run Line Profiling 
First uncomment `@profile` above `blend()` in `blendsql.py`.
Make sure you've run `pip install line_profiler` first. This installs the tool here: https://github.com/pyutils/line_profilerPYTHONPATH=$PWD:$PYTHONPATH kernprof -lv examples/benchmarks/with_blendsql.pyBenchmarksThe below benchmarks were done on my local M1 Macbook Pro. by running the scripts found inexamples/benchmarks.
'Lines of Code' is a rough estimate of the user-written code for each usecase.NameDescriptionRuntime/s (Across 10 runs)Lines of CodeBlendSQL5.685 +/- 0.9309SQL + LLM CallsFiltering what we can with SQL, then running LLM calls.9.083 +/- 2.061106Naive SQL + LLM CallsRuning LLM calls on entire table, regardless of SQL conditions.64.809 +/- 6.225106