Package: annlite
A fast embedded library for approximate nearest neighbor searchWhat is AnnLite?AnnLiteis alightweightandembeddablelibrary forfastandfilterableapproximate nearest neighbor search(ANNS).
It allows to search for nearest neighbors in a dataset of millions of points with a Pythonic API.Highlighted features:🐥Easy-to-use: a simple API is designed to be used with Python. It is easy to use and intuitive to set up to production.🐎Fast: the library uses a highly optimized approximate nearest neighbor search algorithm (HNSW) to search for nearest neighbors.🔎Filterable: the library allows you to search for nearest neighbors within a subset of the dataset.🍱Integration: Smooth integration with neural search ecosystem includingJinaandDocArray,
so that users can easily expose search API withgRPCand/orHTTP.The library is easy to install and use. It is designed to be used with Python.InstallationTo use AnnLite, you need to first install it. The easiest way to install AnnLite is usingpip:pipinstall-Uannliteor install from source:pythonsetup.pyinstallQuick startBefore you start, you need to know some experience aboutDocArray.AnnLiteis designed to be used withDocArray, so you need to know how to useDocArrayfirst.For example, you can create aDocArraywith1000random vectors with128dimensions:fromdocarrayimportDocumentArrayimportnumpyasnpdocs=DocumentArray.empty(1000)docs.embeddings=np.random.random([1000,128]).astype(np.float32)IndexThen you can create anAnnIndexerto index the createddocsand search for nearest neighbors:fromannliteimportAnnLiteann=AnnLite(128,metric='cosine',data_path="/tmp/annlite_data")ann.index(docs)Note that this will create a directory/tmp/annlite_datato persist the documents indexed.
If this directory already exists, the index will be loaded from the directory.
And if you want to create a new index, you can delete the directory first.SearchThen you can search for nearest neighbors for some query docs withann.search():query=DocumentArray.empty(5)query.embeddings=np.random.random([5,128]).astype(np.float32)result=ann.search(query)Then, you can inspect the retrieved docs for each query doc insidequerymatches:forqinquery:print(f'Query{q.id}')fork,minenumerate(q.matches):print(f'{k}:{m.id}{m.scores["cosine"]}')Queryddbae2073416527bad66ff186543eff80:47dcf7f3fdbe3f0b8d73b87d2a1b266f{'value':0.17575037}1:7f2cbb8a6c2a3ec7be024b750964f317{'value':0.17735684}2:2e7eed87f45a87d3c65c306256566abb{'value':0.17917466}Querydda90782f6514ebe4be4705054f744520:6616eecba99bd10d9581d0d5092d59ce{'value':0.14570713}1:d4e3147fc430de1a57c9883615c252c6{'value':0.15338594}2:5c7b8b969d4381f405b8f07bc68f8148{'value':0.15743542}...Or shorten the loop as one-liner using the element & attribute selector:print(query['@m',('id','scores__cosine')])QueryYou can get specific document by its id:doc=ann.get_doc_by_id('<doc_id>')And you can also get the documents withlimitandoffset, which is useful for pagination:docs=ann.get_docs(limit=10,offset=0)Furthermore, you can also get the documents ordered by a specific column from the index:docs=ann.get_docs(limit=10,offset=0,order_by='x',ascending=True)Note: theorder_bycolumn must be one of thecolumnsin the index.UpdateAfter you have indexed thedocs, you can update the docs in the index by callingann.update():updated_docs=docs.sample(10)updated_docs.embeddings=np.random.random([10,128]).astype(np.float32)ann.update(updated_docs)DeleteAnd finally, you can delete the docs from the index by callingann.delete():to_delete=docs.sample(10)ann.delete(to_delete)Search with filtersTo support search with filters, the annlite must be created withcolumsparameter, which is a series of fields you want to filter by.
At the query time, the annlite will filter the dataset by providingconditionsfor certain fields.importannlite# the column schema: (name:str, dtype:type, create_index: bool)ann=annlite.AnnLite(128,columns=[('price',float)],data_path="/tmp/annlite_data")Then you can insert the docs, in which each doc has a fieldpricewith a float value contained in thetags:importrandomdocs=DocumentArray.empty(1000)docs=DocumentArray([Document(id=f'{i}',tags={'price':random.random()})foriinrange(1000)])docs.embeddings=np.random.random([1000,128]).astype(np.float32)ann.index(docs)Then you can search for nearest neighbors with filtering conditions as:query=DocumentArray.empty(5)query.embeddings=np.random.random([5,128]).astype(np.float32)ann.search(query,filter={"price":{"$lte":50}},limit=10)print(f'the result with filtering:')fori,qinenumerate(query):print(f'query [{i}]:')forminq.matches:print(f'\t{m.id}{m.scores["euclidean"].value}(price={m.tags["price"]})')Theconditionsparameter is a dictionary of conditions. The key is the field name, and the value is a dictionary of conditions.
The query language is the same asMongoDB Query Language.
We currently support a subset of those selectors.$eq- Equal to (number, string)$ne- Not equal to (number, string)$gt- Greater than (number)$gte- Greater than or equal to (number)$lt- Less than (number)$lte- Less than or equal to (number)$in- Included in an array$nin- Not included in an arrayThe query will be performed on the field if the condition is satisfied. The following is an example of a query:A Nike shoes with white color{"brand":{"$eq":"Nike"},"category":{"$eq":"Shoes"},"color":{"$eq":"White"}}We also support boolean operators$orand$and:{"$and":{"brand":{"$eq":"Nike"},"category":{"$eq":"Shoes"},"color":{"$eq":"White"}}}A Nike shoes or price less than100$:{"$or":{"brand":{"$eq":"Nike"},"price":{"$lte":100}}}Dump and LoadBy default, the hnsw index is in memory. You can dump the index todata_pathby calling.dump():fromannliteimportAnnLiteann=AnnLite(128,metric='cosine',data_path="/path/to/data_path")ann.index(docs)ann.dump()And you can restore the hnsw index fromdata_pathif it exists:new_ann=AnnLite(128,metric='cosine',data_path="/path/to/data_path")If you didn't dump the hnsw index, the index will be rebuilt from scratch. This will take a while.Supported distance metricsThe annlite supports the following distance metrics:Supported distances:DistanceparameterEquationEuclideaneuclideand = sqrt(sum((Ai-Bi)^2))Inner productinner_productd = 1.0 - sum(Ai*Bi)Cosine similaritycosined = 1.0 - sum(Ai*Bi) / sqrt(sum(Ai*Ai) * sum(Bi*Bi))Note that inner product is not an actual metric. An element can be closer to some other element than to itself.
That allows some speedup if you remove all elements that are not the closest to themselves from the index, e.g.,inner_product([1.0, 1.0], [1.0. 1.0]) < inner_product([1.0, 1.0], [2.0, 2.0])HNSW algorithm parametersThe HNSW algorithm has several parameters that can be tuned to improve the search performance.Search parametersef_search- The size of the dynamic list for the nearest neighbors during search (default:50).
The larger the value, the more accurate the search results, but the slower the search speed.
Theef_searchmust be larger thanlimitparameter insearch(..., limit).limit- The maximum number of results to return (default:10).Construction parametersmax_connection- The number of bi-directional links created for every new element during construction (default:16).
Reasonable range is from2to100. Higher values works better for dataset with higher dimensionality and/or high recall.
This parameter also affects the memory consumption during construction, which is roughlymax_connection * 8-10bytes per stored element.As an example forn_dim=4random vectors optimalmax_connectionfor search is somewhere around6,
while for high dimensional datasets, highermax_connectionare required (e.g.M=48-64) for optimal performance at high recall.
The rangemax_connection=12-48is ok for the most of the use cases.
Whenmax_connectionis changed one has to update the other parameters.
Nonetheless,ef_searchandef_constructionparameters can be roughly estimated by assuming thatmax_connection * ef_{construction}is a constant.ef_construction: The size of the dynamic list for the nearest neighbors during construction (default:200).
Higher values give better accuracy, but increase construction time and memory consumption.
At some point, increasingef_constructiondoes not give any more accuracy.
To setef_constructionto a reasonable value, one can measure the recall: if the recall is lower than 0.9, then increaseef_constructionand re-run the search.To set the parameters, you can define them when creating the annlite:fromannliteimportAnnLiteann=AnnLite(128,columns=[('price',float)],data_path="/tmp/annlite_data",ef_construction=200,max_connection=16)BenchmarkOne can runexecutor/benchmark.pyto get a quick performance overview.Stored dataIndexing timeQuery size=1Query size=8Query size=64100002.9700.0020.0130.10010000076.4740.0110.0780.649500000467.9360.0460.3562.82310000001025.5060.0910.6955.778Results with filtering can be generated fromexamples/benchmark_with_filtering.py. This script should produce a table similar to:Stored data% same filterIndexing timeQuery size=1Query size=8Query size=641000052.8690.0040.0300.27010000152.8690.0040.0350.29410000203.5060.0050.0380.28710000303.5060.0050.0440.35610000503.5060.0080.0640.48410000802.8690.0130.0980.910100000575.9600.0180.1341.0921000001575.9600.0260.2111.7361000002078.4750.0340.2652.0971000003078.4750.0440.3572.8871000005078.4750.0680.5654.3831000008075.9600.1110.8786.8155000005497.7440.0690.5614.43950000015497.7440.1341.0648.46950000020440.1080.1521.1999.47250000030440.1080.2121.65013.26750000050440.1080.3282.63721.96150000080497.7440.5804.60236.986100000051052.3880.1311.0318.2121000000151052.3880.2632.19116.643100000020980.5980.3512.65921.193100000030980.5980.4613.71329.794100000050980.5980.7325.97547.3561000000801052.3881.1519.25573.552Note that:query times presented are represented in seconds.% same filterindicates the amount of data that verifies a filter in the database.For example, if% same filter = 10andStored data = 1_000_000then it means100_000example verify the filter.Next stepsIf you already have experience with Jina and DocArray, you can start usingAnnLiteright away.Otherwise, you can check out this advanced tutorial to learn how to useAnnLite:herein practice.🙋 FAQ1. Why should I useAnnLite?AnnLiteis easy to use and intuitive to set up in production. It is also very fast and memory efficient, making it a great choice for approximate nearest neighbor search.2. How do I useAnnLitewith Jina?We have implemented an executor forAnnLitethat can be used with Jina.fromjinaimportFlowwithFlow().add(uses='jinahub://AnnLiteIndexer',uses_with={'n_dim':128})asf:f.post('/index',inputs=docs)DoesAnnLitesupport search with filters?Yes.DocumentationYou can find the documentation onGithubandReadTheDocs🤝 Contribute and spread the wordWe are also looking for contributors who want to help us improve: code, documentation, issues, feedback! Here is how you can get started:Have a look through GitHub issues labeled "Good first issue".Read our Contributor Covenant Code of ConductOpen an issue or submit your pull request!LicenseAnnLiteis licensed under theApache License 2.0.