Package: openai-to-sqlite
openai-to-sqliteThis tool provides utilities for interacting with OpenAI APIs and storing the results in a SQLite database.SeeSemantic search answers: Q&A against documentation with GPT3 + OpenAI embeddingsfor background on this project.InstallationInstall this tool usingpip:pipinstallopenai-to-sqliteConfigurationYou will need an OpenAI API key to use this tool.You can create one athttps://beta.openai.com/account/api-keysYou can then either set the API key as an environment variable:exportOPENAI_API_KEY=sk-...Or pass it to each command using the--token sk-...option.Calling OpenAI APIs with SQL functionsTheopenai-to-sqlite querycommand can be used to execute SQL queries that call OpenAI APIs.Functions available are:chatgpt(prompt)- call the OpenAI Chat API using modelgpt-3.5-turbowith the specified prompt.chatgpt(prompt, system)- call that API with the prompt and the specified system prompt.More functions are planned in the future.Here's how to use this command to run basic sentiment analysis against content in a table:openai-to-sqlitequerydatabase.db"update messages set sentiment = chatgpt('Sentiment analysis for this message: ' || message ||' - ONLY return a lowercase string from: positive, negative, neutral, unknown')where sentiment not in ('positive', 'negative', 'neutral', 'unknown')or sentiment is null"This updates thesentimentcolumn in a table calledmessages. It populates it with the response from the specified prompt.The command will display a progress bar indicating how many rows are being processed.You can add an emptysentimentcolumn to a table usingsqlite-utilslike this:sqlite-utilsadd-columndatabase.dbmessagessentimentEmbeddingsTheembeddingscommand can be used to calculate and storeOpenAI embeddingsfor strings of text.Each embedding has a cost, so be sure to familiarize yourself withthe pricingfor the embedding model.The command can accept data in four different ways:As a JSON file containing a list of objectsAs a CSV fileAs a TSV fileBy running queries against a SQLite databaseFor all of these formats there should be anidcolumn, followed by one or more text columns.The ID will be stored as the content ID. Any other columns will be concatenated together and used as the text to be embedded.The embeddings from the API will then be saved as binary blobs in theembeddingstable of the specified SQLite database - or another table, if you pass the-t/--tableoption.JSON, CSV and TSVGiven a CSV file like this:id,content
1,This is a test
2,This is another testEmbeddings can be stored like so:openai-to-sqliteembeddingsembeddings.dbdata.csvThe resulting schema looks like this:CREATETABLE[embeddings]([id]TEXTPRIMARYKEY,[embedding]BLOB);The same data can be provided as TSV data:id    content
1     This is a test
2     This is another testThen imported like this:openai-to-sqliteembeddingsembeddings.dbdata.tsvOr as JSON data:[{"id":1,"content":"This is a test"},{"id":2,"content":"This is another test"}]Imported like this:openai-to-sqliteembeddingsembeddings.dbdata.jsonIn each of these cases the tool automatically detects the format of the data. It does this by inspecting the data itself - it does not consider the file extension.If the automatic detection is not working, you can pass--format json,csvortsvto explicitly specify a format:openai-to-sqliteembeddingsembeddings.dbdata.tsv--formattsvImporting data from standard inputYou can use a filename of-to pipe data in to standard input:catdata.tsv|openai-to-sqliteembeddingsembeddings.db-Data from a SQL queryThe--sqloption can be used to read data to be embedded from the attached SQLite database. The query must return anidcolumn and one or more text columns to be embedded.openai-to-sqliteembeddingscontent.db\--sql"select id, title from documents"This will create aembeddingstable in thecontent.dbdatabase and populate it with embeddings calculated from thetitlecolumn in that query.You can also store embeddings in one database while reading data from another database, using the--attach alias filename.dboption:openai-to-sqliteembeddingsembeddings.db\--attachdocumentsdocuments.db\--sql"select id, title from documents.documents"A progress bar will be displayed when using--sqlthat indicates how long the embeddings are likely to take to calculate.The CSV/TSV/JSON options do not correctly display the progress bar. You can work around this by importing your data into SQLite first (e.g.using sqlite-utils) and then running the embeddings using--sql.BatchingEmbeddings will be sent to the OpenAI embeddings API in batches of 100. If you know that your data is short strings you can increase the batch size, up to 2048, using the--batch-sizeoption:openai-to-sqliteembeddingsembeddings.dbdata.csv--batch-size2048Working with the stored embeddingsTheembeddingcolumn is a SQLite blob containing 1536 floating point numbers encoded as a sequence of 4 byte values.You can extract them back to an array of floating point values in Python like this:importstructvector=struct.unpack("f"*1536,binary_embedding)Searching embeddings with the search commandHaving saved the embeddings for content, you can run searches using thesearchcommand:openai-to-sqlitesearchembeddings.db'this is my search term'The output will be a list of cosine similarity scores and content IDs:openai-to-sqlitesearchblog.db'cool datasette demo'0.843 7849
0.830 8036
0.828 8195
0.826 8098
0.818 8086
0.817 8171
0.816 8121
0.815 7860
0.815 7872
0.814 8169Add the-t/--tableoption if your embeddings are stored in a different table:openai-to-sqlitesearchcontent.db'this is my search term'-tdocuments

Add`--count20`toretrieve20results(thedefaultis10).Search for similar content with the similar commandHaving saved the embeddings for content, you can search for similar content with thesimilarcommand:oopenai-to-sqlitesimilarembeddings.db'<content identifier>'The output will be a list of cosine similarity scores and content IDs:openai-to-sqlitesimilarembeddings-bjcp-2021.db'23G Gose'23G Gose
  1.000 23G Gose
  0.929 24A Witbier
  0.921 23A Berliner Weisse
  0.909 05B Kölsch
  0.907 01D American Wheat Beer
  0.906 27 Historical Beer: Lichtenhainer
  0.905 23D Lambic
  0.905 10A Weissbier
  0.904 04B Festbier
  0.904 01B American LagerYou can pass more than one IDs to see similarities calculated for each one:openai-to-sqlitesimilarembeddings-bjcp-2021.db\'23G Gose''01A American Light Lager'Or pass--allto run similarity for every item in the database. This runs similarity calculations for the number of items squared so it can be quite a long running operation:openai-to-sqlitesimilarembeddings-bjcp-2021.db--allTo save these calculations to asimilaritiestable in the database, use the--saveoption:openai-to-sqlitesimilarembeddings-bjcp-2021.db--all--saveThe--saveoption disables output. You can re-enable output with--print:openai-to-sqlitesimilarembeddings-bjcp-2021.db--all--save--printTo save to a database table with a name other thansimilarities, use--table:openai-to-sqlitesimilarembeddings-bjcp-2021.db\--all--save--tablemy_similarities--recalculate-for-matchesRe-calculating similarities for every row in the database can be quite a lengthy operation.If you know which rows have just been added, you can speed things up using--recalculate-for-matches.This tellsopenai-to-sqlite similarto only re-calculate similarities for rows that are close matches to the specified rows.This means you can add one or two additional records and then trigger an update of the saved similarity scores for just those new records plus for the twenty closest matches to those new records like this:openai-to-sqlitesimilarembeddings-bjcp-2021.db\--save'23G Gose''01A American Light Lager'\--recalculate-for-matches\--count20\--printDevelopmentTo contribute to this tool, first checkout the code. Then create a new virtual environment:cdopenai-to-sqlite
python-mvenvvenvsourcevenv/bin/activateNow install the dependencies and test dependencies:pipinstall-e'.[test]'To run the tests:pytest