Package: datasette-faiss
datasette-faissMaintain aFAISS indexfor specified Datasette tablesSeeSemantic search answers: Q&A against documentation with GPT3 + OpenAI embeddingsfor background on this project.InstallationInstall this plugin in the same environment as Datasette.datasette install datasette-faissUsageThis plugin creates in-memory FAISS indexes for specified tables on startup, using anIndexFlatL2FAISS index type.If the tables are modified after the server has started the indexes will not (yet) pick up those changes.ConfigurationThe tables to be indexed must haveidandembeddingcolumns. Theembeddingcolumn must be ablobcontaining embeddings that are arrays of floating point numbers that have been encoded using the following Python function:defencode(vector):returnstruct.pack("f"*len(vector),*vector)You can import that function from this package like so:fromdatasette_faissimportencodeYou can specify which tables should have indexes created for them by adding this tometadata.json:{"plugins":{"datasette-faiss":{"tables":[["blog","embeddings"]]}}}Each table is an array listing the database name and the table name.If you are usingmetadata.ymlthe configuration should look like this:plugins:datasette-faiss:tables:-["blog","embeddings"]SQL functionsThe plugin makes four new SQL functions available within Datasette:faiss_search(database, table, embedding, k)Returns theknearest neighbors to theembeddingfound in the specified database and table. For example:selectfaiss_search('blog','embeddings',(selectembeddingfromembeddingswhereid=3),5)This will return a JSON array of the five IDs of the records in theembeddingstable in theblogdatabase that are closest to the specified embedding. The returned value looks like this:["1","1249","1011","5","10"]You can use the SQLitejson_each()function to turn that into a table-like sequence that you can join against.Here's an example query that does that:withrelatedas(selectvaluefromjson_each(faiss_search('blog','embeddings',(selectembeddingfromembeddingslimit1),5)))select*fromblog_entry,relatedwhereid=valuefaiss_search_with_scores(database, table, embedding, k)Takes the same arguments as above, but the return value is a JSON array of pairs, each with an ID and a score - something like this:[["1",0.0],["1249",0.21042244136333466],["1011",0.29391372203826904],["5",0.29505783319473267],["10",0.31554925441741943]]faiss_encode(json_vector)Given a JSON array of floats, returns the binary embedding blob that can be used with the other functions:selectfaiss_encode('[2.4, 4.1, 1.8]')-- Returns a 12 byte blobselecthex(faiss_encode('[2.4, 4.1, 1.8]'))-- Returns 9A991940333383406666E63Ffaiss_decode(vector_blob)The opposite offaiss_encode().selectfaiss_decode(X'9A991940333383406666E63F')Returns:[2.4000000953674316,4.099999904632568,1.7999999523162842]Note that floating point arithmetic results in numbers that don't quite round-trip to the exact same expected value.faiss_agg(id, embedding, compare_embedding, k)This aggregate function can be used to find theknearest neighbors tocompare_embeddingfor each unique value ofidin the table. For example:selectfaiss_agg(id,embedding,(selectembeddingfromembeddingswhereid=3),5)fromembeddingsUnlike thefaiss_search()function, this does not depend on the per-table index that the plugin creates when it first starts running. Instead, an index is built every time the aggregation function is run.This means that it should only be used on smaller sets of values - once you get above 10,000 or so the performance from this function is likely to become prohibitively expensive.The function returns a JSON array of IDs representing thekrows with the closest distance scores, like this:[1324,344,5562,553,2534]You can use thejson_each()function to turn that into a table-like sequence that you can join against.faiss_agg_with_scores(id, embedding, compare_embedding, k)This is similar to thefaiss_agg()aggregate function but it returns a list of pairs, each with an ID and the corresponding score - something that looks like this (ifkwas 2):[[2412,0.25],[1245,24.25]]DevelopmentTo set up this plugin locally, first checkout the code. Then create a new virtual environment:cd datasette-faiss
python3 -m venv venv
source venv/bin/activateNow install the dependencies and test dependencies:pip install -e '.[test]'To run the tests:pytest