Package: pqlite
PQLitePQLiteis anApproximate Nearest Neighbor Search(ANNS) library integrated with the Jina ecosystem.This indexer is recommended to be used when an application requiressearch with filtersapplied onDocumenttags.
Thefiltering query languageis based onMongoDB's query and projection operators. We currently support a subset of those selectors.
The tags filters can be combined with$andand$or:$eq- Equal to (number, string)$ne- Not equal to (number, string)$gt- Greater than (number)$gte- Greater than or equal to (number)$lt- Less than (number)$lte- Less than or equal to (number)$in- Included in an array$nin- Not included in an arrayFor example, we want to search for a product with a price no more than50$.index.search(query,filter={"price":{"$lte":50}})More example filter expressesA Nike shoes with white color{"brand":{"$eq":"Nike"},"category":{"$eq":"Shoes"},"color":{"$eq":"White"}}Or{"$and":{"brand":{"$eq":"Nike"},"category":{"$eq":"Shoes"},"color":{"$eq":"White"}}}A Nike shoes or price less than100${"$or":{"brand":{"$eq":"Nike"},"price":{"$lt":100}}}InstallationTo install PQLite you can simply run:pipinstallhttps://github.com/jina-ai/pqlite/archive/refs/heads/main.zipGetting StartedFor an in-depth overview of the features of PQLite
you can follow along with one of the examples below:NameLinkE-commerce product image search with PQLiteQuick StartCreate a newpqliteimportrandomimportnumpyasnpfromjinaimportDocument,DocumentArrayfrompqliteimportPQLiteN=10000# number of data pointsNq=10# number of query dataD=128# dimentionality / number of features# the column schema: (name:str, dtype:type, create_index: bool)pqlite=PQLite(dim=D,columns=[('price',float)],data_path='./workspace_data')Note that this will create a folder./workspace_datawhere indexed data will be stored.
If there is already a folder with this name and the code presented here is not working remove that folder.Add new dataX=np.random.random((N,D)).astype(np.float32)# 10,000 128-dim vectors to be indexeddocs=DocumentArray([Document(id=f'{i}',embedding=X[i],tags={'price':random.random()})foriinrange(N)])pqlite.index(docs)Search with filteringXq=np.random.random((Nq,D)).astype(np.float32)# a 128-dim query vectorquery=DocumentArray([Document(embedding=Xq[i])foriinrange(Nq)])# without filteringpqlite.search(query,limit=10)print(f'the result without filtering:')fori,qinenumerate(query):print(f'query [{i}]:')forminq.matches:print(f'\t{m.id}({m.scores["euclidean"].value})')# with filteringpqlite.search(query,filter={"price":{"$lte":50}},limit=10)print(f'the result with filtering:')fori,qinenumerate(query):print(f'query [{i}]:')forminq.matches:print(f'\t{m.id}{m.scores["euclidean"].value}(price={m.tags["x"]})')Update dataXn=np.random.random((10,D)).astype(np.float32)# 10,000 128-dim vectors to be indexeddocs=DocumentArray([Document(id=f'{i}',embedding=Xn[i],tags={'price':random.random()})foriinrange(10)])pqlite.update(docs)Delete datapqlite.delete(['1','2'])BenchmarkOne can runexecutor/benchmark.pyto get a quick performance overview.Stored dataIndexing timeQuery size=1Query size=8Query size=64100002.9700.0020.0130.10010000076.4740.0110.0780.649500000467.9360.0460.3562.82310000001025.5060.0910.6955.778Results with filtering can be generated fromexamples/benchmark_with_filtering.py. This script should produce a table similar to:Stored data% same filterIndexing timeQuery size=1Query size=8Query size=641000052.8690.0040.0300.27010000152.8690.0040.0350.29410000203.5060.0050.0380.28710000303.5060.0050.0440.35610000503.5060.0080.0640.48410000802.8690.0130.0980.910100000575.9600.0180.1341.0921000001575.9600.0260.2111.7361000002078.4750.0340.2652.0971000003078.4750.0440.3572.8871000005078.4750.0680.5654.3831000008075.9600.1110.8786.8155000005497.7440.0690.5614.43950000015497.7440.1341.0648.46950000020440.1080.1521.1999.47250000030440.1080.2121.65013.26750000050440.1080.3282.63721.96150000080497.7440.5804.60236.986100000051052.3880.1311.0318.2121000000151052.3880.2632.19116.643100000020980.5980.3512.65921.193100000030980.5980.4613.71329.794100000050980.5980.7325.97547.3561000000801052.3881.1519.25573.552Note that:query times presented are represented in seconds.% same filterindicates the amount of data that verifies a filter in the database.For example, if% same filter = 10andStored data = 1_000_000then it means100_000example verify the filter.Implemented AlgorithmsCurrently PQLite supports:HNSW Algorithm (default choice)PQ-linear-scan (requires training)Research foundations of PQLiteXor FiltersFaster and Smaller Than Bloom FiltersCVPR20 TutorialBillion-scale Approximate Nearest Neighbor SearchXOR-QuantizationFast top-K Cosine Similarity Search through XOR-Friendly Binary Quantization on GPUsNeurIPS21 ChallengeBillion-Scale Approximate Nearest Neighbor Search ChallengeNeurIPS'21 competition trackPAMI 2011Product quantization for nearest neighbor searchCVPR 2016Efficient Indexing of Billion-Scale Datasets of Deep DescriptorsNIPs 2017Multiscale Quantization for Fast Similarity SearchNIPs 2018Non-metric Similarity Graphs for Maximum Inner Product SearchACMMM 2018Reconfigurable Inverted IndexcodeECCV 2018Revisiting the Inverted Indices for Billion-Scale Approximate Nearest NeighborsCVPR 2019Unsupervised Neural Quantization for Compressed-Domain Similarity SearchICML 2019Learning to Route in Similarity GraphsICML 2020Graph-based Nearest Neighbor Search: From Practice to Theory