Package: pymilvusdm
Milvus Data Migration ToolOverviewMilvusdm (Milvus Data Migration) is a data migration tool forMilvusthat supports importing Faiss and HDF5 data files into Milvus, migrating data between Milvus, and it also supports batch backup of Milvus data to local files. Using milvusdm can help developers improve usage efficiency, reduce operation and maintenance costs.Getting startedPrerequisitesOperating system requirementsOperating systemSupported versionsCentOS7.5 or higherUbuntu LTS18.04 or higherSoftware requirementsSoftwareVersionMilvus0.10.x or 1.x or 2.xPython33.7 or higherpip3Corresponds to python version.InstallConfigure environment variablesAdd the following two lines to~/.bashrcfile:exportMILVUSDM_PATH='/home/$user/milvusdm'exportLOGS_NUM=0MILVUSDM_PATH: This parameter defines the working path of milvusdm. Logs and data generated by Milvusdm will be stored in this path.  The default value is/home/$user/milvusdm.LOGS_NUM： Milvusdm log file generates one per day. This parameter defines the number of log files to be saved. The default value is 0, which means all log files are saved.Make the configured environment variables：$source~/.bashrcInstall milvusdm by pip$pip3installpymilvusdm==2.0The pymilvusdm2.0 is used to migrate datafrom Milvus(0.10.x or 1.x) to Milvus2.x.How to useFaiss to MilvusExport one Faiss index file to Milvus in specified collection or partition.In the current version, only flat and ivf_flat indexes for floating data are supported.Download the example yaml$wgethttps://raw.githubusercontent.com/milvus-io/milvus-tools/main/yamls/F2M.yamlConfig the yamlF2M:milvus_version:1.xdata_path:'/home/data/faiss1.index'dest_host:'127.0.0.1'dest_port:19530mode:'append'dest_collection_name:'test'dest_partition_name:''collection_parameter:dimension:256index_file_size:1024metric_type:'L2'Optional parameters:dest_partition_nameThe  Parametermodecan be selected fromappend,skip,overwrite.  This parameter takes effect only when the specified collection name exists in Milvus library.​       append: Append data to the existing collection​       skip: Skip the existing collection and do not perform any operations​       overwrite: Delete the old collection, create a new collection with the same name and then import the data.Parameter descriptionparameterdescriptionexampleF2MTask: Export data in HDF5  to Milvus.milvus_versionVersion of Milvus.0.10.5data_pathThe path of fairs data'/home/user/data/faiss.index'dest_hostHost of Milvus.'127.0.0.1'dest_portPort of Milvus.19530modeThe mode for importing data to Milvus.'append'dest_collection_nameCollection to be imported fairs data.'test'dest_partition_namePartition to be imported fairs data. (Optional)'partition'collection_parameterParameters used to create the collection.dimension: 512index_file_size: 1024metric_type: 'HAMMING'Usage$milvusdm--yamlF2M.yamlHDF5 to MilvusExport one or more HDF5 file to Milvus in specified collection or partition.We provide the HDF5 examples offloat vectors(dim-100) andbinary vectors(dim-512) and their corresponding ids.Download the yaml$wgethttps://raw.githubusercontent.com/milvus-io/milvus-tools/main/yamls/H2M.yamlConfig the yamlH2M:milvus-version:1.xdata_path:-/Users/zilliz/float_1.h5-/Users/zilliz/float_2.h5data_dir:dest_host:'127.0.0.1'dest_port:19530mode:'overwrite'# 'skip/append/overwrite'dest_collection_name:'test_float'dest_partition_name:'partition_1'collection_parameter:dimension:128index_file_size:1024metric_type:'L2'orH2M:milvus_version:1.xdata_path:data_dir:'/Users/zilliz/HDF5_data'dest_host:'127.0.0.1'dest_port:19530mode:'append'# 'skip/append/overwrite'dest_collection_name:'test_binary'dest_partition_name:collection_parameter:dimension:512index_file_size:1024metric_type:'HAMMING'Optional parameters:dest_partition_nameJust configuredata_pathordata_dir, while the other one is None.Parameter descriptionparameterdescriptionexampleH2MTask: Export data in HDF5  to Milvus.milvus_versionVersion of Milvus.0.10.5data_pathList of HDF5 files.- /Users/zilliz/float_1.h5- /Users/zilliz/float_1.h5data_dirPath of directory containing HDF5 files./Users/zilliz/Desktop/HDF5_datadest_hostHost of Milvus.'127.0.0.1'dest_portPort of Milvus.19530modeThe mode for importing data to Milvus.'append'dest_collection_nameCollection to be imported HDF5 data.'test_float'dest_partition_namePartition to be imported HDF5 data.(optional)'partition_1'collection_parameterParameters used to create the collection.dimension: 512index_file_size: 1024metric_type: 'HAMMING'Usage$milvusdm--yamlH2M.yamlMilvus to MilvusCopy a collection of source_milvus or multiple partitions of a collection into the corresponding collection or partition in dest_milvus.Download the yaml$wgethttps://raw.githubusercontent.com/milvus-io/milvus-tools/main/yamls/M2M.yamlConfig the yamlM2M:milvus_version:1.xsource_milvus_path:'/home/user/milvus'mysql_parameter:host:'127.0.0.1'user:'root'port:3306password:'123456'database:'milvus'source_collection:# specify the 'partition_1' and 'partition_2' partitions of the 'test' collection.test:-'partition_1'-'partition_2'dest_host:'127.0.0.1'dest_port:19530mode:'skip'# 'skip/append/overwrite'OrM2M:milvus_version:1.xsource_milvus_path:'/home/user/milvus'mysql_parameter:source_collection:# specify the collection named 'test'test:dest_host:'127.0.0.1'dest_port:19530mode:'skip'# 'skip/append/overwrite'Required parameters:source_milvus_path,source_collection,dest_host,dest_portandmode.If you are using MySQL to manage source_milvus metadata, configure themysql_parameterparameter, which is empty if you are using SQLite.Thesource_collectionparameter must specify a collection name, and the following partition name is optional and multiple partitions can be added.Parameter DescriptionparameterdescriptionexampleM2MTask: Copy the data from Milvus to the same version of Milvus.milvus_versionThe dest-milvus version.0.10.5source_milvus_pathThe local path of the Milvus container to be mounted.'/home/user/milvus'mysql_parameterUse Mysql for metadata management related parameters, including mysqlhost,user,port,passwordanddatabaseparameters.host: '127.0.0.1'user: 'root'port: 3306password: '123456'database: 'milvus'source_collectionThe name of the collection or partition specified when exporting Milvus data.test:- 'partition_1'- 'partition_2'dest_hostDestination Milvus port.'127.0.0.1'dest_portDestination Milvus port.19530modeThe mode for importing data to destination Milvus.'skip'UsageIt will copy thesource_milvuscollection data todest_milvus.$milvusdm--yamlM2M.yamlMilvus to HDF5Export a Milvus collection or multiple partitions of a collection to a local HDF5 format file.Download the yaml$wgethttps://raw.githubusercontent.com/milvus-io/milvus-tools/main/yamls/M2H.yamlConfig the yamlM2H:milvus_version:1.xsource_milvus_path:'/home/user/milvus'mysql_parameter:host:'127.0.0.1'user:'root'port:3306password:'123456'database:'milvus'source_collection:# specify the 'partition_1' and 'partition_2' partitions of the 'test' collection.test:-'partition_1'-'partition_2'data_dir:'/home/user/data'OrM2H:milvus_version:1.xsource_milvus_path:'/home/user/milvus'mysql_parameter:source_collection:# specify the collection named 'test'test:data_dir:'/home/user/data'Thesource_milvus_path,source_collection, anddata_dirparameters are required.If you are using MySQL to manage source_milvus metadata, configure themysql_parameterparameter, which is empty if you are using SQLite.Thesource_collectionparameter must specify a collection name, and the following partition name is optional and multiple partitions can be added.Parameter descriptionparameterdescriptionexampleM2HTask: Export Milvus data to local HDF5 format files.milvus_versionThe source-milvus version.0.10.5source_milvus_pathThe local path of the Milvus container to be mounted.'/home/user/milvus'mysql_parameterUse Mysql for metadata management related parameters, including mysqlhost,user,port,passwordanddatabaseparameters.host: '127.0.0.1'user: 'root'port: 3306password: '123456'database: 'milvus'source_collectionThe name of the collection or partition specified when exporting Milvus data.test:- 'partition_1'- 'partition_2'data_dirLocal path of the hdf5 format file.'/home/user/data'UsageIt will generate the corresponding hfd5 format file and H2M configuration file in thedata_dirdirectory.$milvusdm--yamlM2H.yamlCode StructureIf you would like to contribute code to this project, you can find out more about our code structure:pymilvusdmcoremilvus_client.py, the function about Milvusread_data.py, read local data files in HDF5 format (if you need to read files in other formats, you can write the code here)read_faiss_data.py, read the Faiss data fileread_milvus_data.py, read the Milvus data fileread_milvus_meta.py, read Milvus' meta datadata_to_milvus.py, create collections or partitions based on the parameters of the yaml file and import the vectors and ids into Milvus.save_data.py, save the data as a file in HDF5 formatwrite_logs.py, write debug/info/error logs with an operationfaiss_to_milvus.py, import of Faiss file data into Milvushdf5_to_milvus.py, import file data in HDF5 format into Milvusmilvus_to_milvus.py，copy data from Milvus to another Milvusmilvus_to_hdf5.py, export Milvus data to HDF5 format filesmain.py, read yaml files and do related tasks.setting.py, relevant configuration parameters when executing the codesetup.py, package and upload pymilvusdm to PypiTodoFaiss to MilvusSupports vector data of binary type.Milvus to MilvusSpecify multiple collections or multiple partitions based on blocklist and allowlist.Whensource_collection='*', all Milvus data is exported.Support for merging multiple collections or partitions in source_milvus into one collection in dest_milvus.Milvud DumpMilvus Restore