Package: pymilvus-distributed
Milvus Python SDKPython SDK forMilvus. To contribute code to this project, please read ourcontribution guidelinesfirst.For detailed SDK documentation, refer toAPI Documentation.New featuresGet startedPrerequisitesInstall pymilvusExamplesBasic operationsConnect to the Milvus serverCreate/Drop collectionsCreate a collectionDrop a collectionCreate/Drop partitions in a collectionCreate a partitionDrop a partitionCreate/Drop indexes in a collectionCreate an indexDrop an indexInsert/Delete vectors in collections/partitionsInsert entities in a collectionInsert entities in a partitionDelete entities by IDFlush data in one or multiple collections to diskCompact all segments in a collectionSearch entities in collections/partitionsSearch entities in a collectionSearch entities in a partitionDisconnect from the Milvus serverFAQNew featuresremoveget_index_info, the index info can be obtained byget_collection_infoaddget_collection_stats, more detailed collection stats.Get startedPrerequisitespymilvus only supports Python 3.5 or higher.Install pymilvusYou can install pymilvus viapiporpip3for Python3:$pip3installpymilvusThe following collection shows Milvus versions and recommended pymilvus versions:Milvus versionRecommended pymilvus version0.3.00.1.130.3.10.1.250.4.00.2.20.5.00.2.30.5.10.2.30.5.20.2.30.5.30.2.50.6.00.2.6, 0.2.70.7.00.2.80.7.10.2.90.8.00.2.100.9.00.2.110.9.10.2.120.10.00.2.13>=0.10.1, <0.11.00.2.140.11.00.3.0You can install a specific version of pymilvus by:$pipinstallpymilvus==0.3.0You can upgradepymilvusto the latest version by:$pipinstall--upgradepymilvusExamplesRefer toexamplesfor more example programs.Basic operationsConnect to the Milvus serverImport pymilvus.# Import pymilvus>>>frommilvusimportMilvus,DataTypeCreate a client to Milvus server using one of the following methods:# Connect to Milvus server>>>client=Milvus(host='localhost',port='19530')Note: In the above code, default values are used forhostandportparameters.
Feel free to change them to the IP address and port you set for Milvus server.>>>client=Milvus(uri='tcp://localhost:19530')Create/Drop collectionsCreate a collectionPrepare collection parameters.# create collection name>>>collection_name='test01'# create a collection of 4 fields, fields A, B and C are int type fields# and Vec is a float vector field.# segment_row_limit is default as 524288 if not specified>>>collection_param={..."fields":[...{"name":"A","type":DataType.INT32},...{"name":"B","type":DataType.INT32},...{"name":"C","type":DataType.INT64},...{"name":"Vec","type":DataType.FLOAT_VECTOR,"params":{"dim":128}}...],..."segment_row_limit":4096,..."auto_id":True...}Create collectiontest01with dimension of 128, size of the data file for Milvus to automatically
create indexes as 4096. Ifmetric_typeisn't offered, default metric type is Euclidean distance (L2).
ForFLOAT_VECTORfield,dimis a must.# Create a collection>>>client.create_collection(collection_name,collection_param)You can check collection info byget_collection_info>>>info=client.get_collection_info('test01')>>>info{'fields':[{'name':'A','type':<DataType.INT32:4>,'params':{},'indexes':[{}]},{'name':'C','type':<DataType.INT64:5>,'params':{},'indexes':[{}]},{'name':'B','type':<DataType.INT32:4>,'params':{},'indexes':[{}]},{'name':'Vec','type':<DataType.FLOAT_VECTOR:101>,'params':{'dim':128},'indexes':[{}]}],'auto_id':True,'segment_row_limit':4096}You can see from the info, there is anauto_idoption in collection info, and itsTrueby default.
So if you have your own ids and don't want auto generated ids, you may want to setauto_idtoFalsewhile creating collections.Drop a collection# Drop collection>>>status=client.drop_collection('test01')>>>statusStatus(code=0,message='OK')Create/Drop partitions in a collectionCreate a partitionYou can split collections into partitions by partition tags for improved search performance.# Create partition>>>client.create_partition(collection_name='test01',partition_tag='tag01')Uselist_partitions()to verify whether the partition is created.# Show partitions>>>partitions=client.list_partitions(collection_name='test01')>>>partitions['_default','tag01']Drop a partition# Drop partitions>>>status=client.drop_partition(collection_name='test01',partition_tag='tag01')Status(code=0,message='OK')Create/Drop indexes in a collectionCreate an indexNote: In production, it is recommended to create indexes before inserting vectors into the collection.
Index is automatically built when vectors are being imported. However, you need to create the same index
again after the vector insertion process is completed because some data files may not meet theindex_file_sizeand index will not be automatically built for these data files.Create an index ofIVF_FLATwithnlist = 100for the collection.# Create index>>>status=client.create_index('test01',"Vec",{"index_type":"IVF_FLAT","metric_type":"L2","params":{"nlist":100}})>>>statusStatus(code=0,message='OK')You can check index info byget_collection_info>>>info=client.get_collection_info('test01')>>>info{'fields':[{'name':'A','type':<DataType.INT32:4>,'params':{},'indexes':[{}]},{'name':'C','type':<DataType.INT64:5>,'params':{},'indexes':[{}]},{'name':'B','type':<DataType.INT32:4>,'params':{},'indexes':[{}]},{'name':'Vec','type':<DataType.FLOAT_VECTOR:101>,'params':{'dim':128,'metric_type':'L2'},'indexes':[{'index_type':'IVF_FLAT','metric_type':'L2','params':{'nlist':100}}]}],'auto_id':True,'segment_row_limit':4096}Drop an index# Drop an index of a specific field "Vec">>>status=client.drop_index('test01',"Vec")Status(code=0,message='OK')Insert/Delete entities in collections/partitionsInsert entities in a collectionGenerate 5000 vectors of 128 dimension and an integer list.>>>importrandom>>>num=5000# Generate a list of integer.>>>list_of_int=[random.randint(0,255)for_inrange(num)]# Generate 20 vectors of 128 dimension>>>vectors=[[random.random()for_inrange(128)]for_inrange(num)]Create hybrid entities>>>hybrid_entities=[{"name":"A","values":list_of_int,"type":DataType.INT32},{"name":"B","values":list_of_int,"type":DataType.INT32},{"name":"C","values":list_of_int,"type":DataType.INT64},{"name":"Vec","values":vectors,"type":DataType.FLOAT_VECTOR}]Insert the hybrid entities.If you create a new collection withauto_id = True, Milvus automatically generates IDs for the vectors.# Insert vectors>>>ids=client.insert('test01',hybrid_entities)If you create a new collection withauto_id = False, you have to provide user-defined vector ids:# Generate fake custom ids>>>vector_ids=[idforidinrange(num)]# Insert to the non-auto-id collection>>>ids=client.insert('test01',hybrid_entities,ids=vector_ids)The examples below are based on collection withauto_id = True.Insert entities in a partition>>>inserted_vector_ids=client.insert('test01',hybrid_entities,partition_tag="tag01")To verify the entities you have inserted, useget_entity_by_id().>>>entities=client.get_entity_by_id(collection_name='test01',ids=inserted_vector_ids[:10])Delete entities by IDYou can delete these entities by:>>>status=client.delete_entity_by_id('test01',ids[:10])>>>statusStatus(code=0,message='OK')Flush data in one or multiple collections to diskWhen performing operations related to data changes, you can flush the data from memory to disk to avoid possible data loss. Milvus also supports automatic flushing, which runs at a fixed interval to flush the data in all collections to disk. You can use theMilvus server configuration fileto set the interval.>>>client.flush(['test01'])Compact all segments in a collectionA segment is a data file that Milvus automatically creates by merging inserted vector data. A collection can contain multiple segments. If some vectors are deleted from a segment, the space taken by the deleted vectors cannot be released automatically. You can compact segments in a collection to release space.>>>status=client.compact('test01')>>>statusStatus(code=0,message='OK')Search entities in collections/partitionsSearch entities in a collectionPrepare search parameters."term"and"range"is optional,"params"in"vector"stands for index params.# This dsl will search topk `entities` that are# close to vectors[:1] searched by `IVF_FLAT` index with `nprobe = 10` and `metric_type = L2`,# AND field "A" in [1, 2, 5],# AND field "B" greater than 1 less than 100>>>dsl={..."bool":{..."must":[...{..."term":{"A":[1,2,5]}...},...{..."range":{"B":{"GT":1,"LT":100}}...},...{..."vector":{..."Vec":{"topk":10,"query":vectors[:1],"metric_type":"L2","params":{"nprobe":10}}...}...}...]...}...}A search without hybrid conditions withIVF_FLATindex would be like:>>>dsl={..."bool":{..."must":[...{..."vector":{..."Vec":{"topk":10,"query":vectors[:1],"metric_type":"L2","params":{"nprobe":10}}...}...}...]...}...}AFLATsearch doesn't need index params, so the query would be like:>>>dsl={..."bool":{..."must":[...{..."vector":{..."Vec":{"topk":10,"query":vectors[0],"metric_type":"L2"}...}...}...]...}...}Search entities.Withfields=["B"], not only can you get entity ids and distances, but also values of a spacific field B.# search entities and get entity field B back>>>results=client.search('test01',dsl,fields=["B"])You can obtain ids, distances and fields by entities in results.# Results consist of number-of-query entities>>>entities=results[0]# Entities consists of topk entity>>>entity=entities[0]# You can get all ids and distances by entities>>>all_ids=entities.ids>>>all_distances=entities.distances# Or you can get them one by one by entity>>>a_id=entity.id>>>a_distance=entity.distance>>>a_field=entity.entity.B# getattr(entity.entity, "B")Note: If you don't provide fields in search, you will only get ids and distances.Search entities in a partition# Search entities in a partition `tag01`>>>client.search(collection_name='test01',dsl=dsl,partition_tags=['tag01'])Note: If you do not specifypartition_tags, Milvus searches the whole collection.Close client>>>client.close()FAQI'm getting random "socket operation on non-socket" errors from gRPC when connecting to Milvus from an application served on GunicornMake sure to set the environment variableGRPC_ENABLE_FORK_SUPPORT=1. For reference, seehttps://zhuanlan.zhihu.com/p/136619485