Package: dataverse-sdk
Dataverse SDK For PythonDataverse is a MLOPs platform for assisting in data selection, data visualization and model training in computer vision.
Use Dataverse-SDK for Python to help you to interact with the Dataverse platform by Python. Currently, the library supports:Create Project with your input ontology and sensorsGet Project by project-idCreate Dataset from your AWS/Azure storage or localGet Dataset by dataset-idList models for your selected project-idGet and download your modelPackage (PyPi)|Source codeGetting startedInstall the packagepip install dataverse-sdkPrerequisites: You must have an Dataverse Platform Account andPython 3.9+to use this package.Create the clientInteraction with the Dataverse site starts with an instance of theDataverseClientclass. You need site url, an email-account and its password to instantiate the client object.fromdataverse_sdkimport*fromdataverse_sdk.connectionsimportget_connectionclient=DataverseClient(host=DataverseHost.PRODUCTION,email="XXX",password="***",alias="default",force=False,)assertclientisget_connection()# Should provide different alias if you are trying to connect to different workspacesclient2=DataverseClient(host=DataverseHost.PRODUCTION,email="account-2",password="***",alias="client2",force=False,)assertclient2isget_connection()Input arguments:Argument nameType/OptionsDefaultDescriptionhoststr＊--the host url of the dataverse siteemailstr＊--the email account of your dataverse workspacepasswordstr＊--the password of your dataverse workspacealiasstr'default'the connection alias of your dataverse clientforceboolFalsewhether force to replace the connection if the given alias existsKey conceptsOnce you've initialized a DataverseClient, you can interact with Dataverse from the initialized object.ExamplesThe following sections provide examples for the most common DataVerse tasksm including:List ProjectsCreate ProjectGet ProjectCreate DatasetGet DatasetList ModelsGet and Download ModelList ProjectsThelist_projectsmethod will list all projects of the given sites.Example Usage:projects=client.list_projects(current_user=True,exclude_sensor_type=SensorType.LIDAR,image_type=OntologyImageType._2D_BOUNDING_BOX)Input arguments:Argument nameType/OptionsDefaultDescriptioncurrent_userboolTrueonly show the projects of current userexclude_sensor_typeSensorType.CAMERASensorType.LIDARNoneexclude the projects with the given sensor typeimage_typeOntologyImageType._2D_BOUNDING_BOXOntologyImageType.SEMANTIC_SEGMENTATIONOntologyImageType.CLASSIFICATIONOntologyImageType.POINTOntologyImageType.POLYGONOntologyImageType.POLYLINENoneonly include the projects with the given image typeCreate ProjectThecreate_projectmethod will create project on the connected site with the defined ontology and sensors.Example Usage:# 1) Create ontology with ontologyclass objectontology=Ontology(name="sample ontology",image_type=OntologyImageType._2D_BOUNDING_BOX,pcd_type=None,classes=[OntologyClass(name="Pedestrian",rank=1,color="#234567"),OntologyClass(name="Truck",rank=2,color="#345678"),OntologyClass(name="Car",rank=3,color="#456789"),OntologyClass(name="Cyclist",rank=4,color="#567890"),OntologyClass(name="DontCare",rank=5,color="#6789AB"),OntologyClass(name="Misc",rank=6,color="#789AB1"),OntologyClass(name="Van",rank=7,color="#89AB12"),OntologyClass(name="Tram",rank=8,color="#9AB123"),OntologyClass(name="Person_sitting",rank=9,color="#AB1234"),],)For project with camera sensor, there would be only one image_type for one project. You could choose from[OntologyImageType._2D_BOUNDING_BOX, OntologyImageType.SEMANTIC_SEGMENTATION, OntologyImageType.CLASSIFICATION, OntologyImageType.POINT, OntologyImageType.POLYGON, OntologyImageType.POLYLINE].For project with lidar sensor, your should assignpcd_type = OntologyPcdType.CUBOIDfor the ontology.# 2) Create your sensor list with name / SensorTypesensors=[Sensor(name="camera1",type=SensorType.CAMERA),Sensor(name="lidar1",type=SensorType.LIDAR),]# 3) Create your project tag attributes (Optional)project_tag=ProjectTag(attributes=[{"name":"year","type":"number"},{"name":"unknown_object","type":"option","options":[{"value":"fire"},{"value":"leaves"},{"value":"water"}],},])# 4) Create your project with your ontology/sensors/project_tagproject=client.create_project(name="Sample project",ontology=ontology,sensors=sensors,project_tag=project_tag)Input arguments for creating project:Argument nameType/OptionsDefaultDescriptionnamestr*--name of your projectontologyOntology*--the Ontology basemodel data of current projectsensorslist[Sensor]*--the list of Sensor basemodel data of your projectproject_tagProjectTagNoneyour project tagsdescriptionstrNoneyour project description＊--: required argument without defaultCheckhttps://linkervision.gitbook.io/dataverse/data-management/project-ontologyfor the detail ofProject OntologyGet ProjectTheget_proejctmethod retrieves the project from the connected site. Theproject_idparameter is the unique integer ID of the project, not its "name" property.project=client.get_project(project_id=1,client_alias=client.alias)# if client_alias is not provided, we'll get it from clientEdit ProjectFor editing project contents, we have four functions below for add/edit project tag and ontology classes.Add New Project TagsNote: Can not create existing project tag!tag={"attributes":[{"name":"month","type":"number"},{"name":"weather","type":"option","options":[{"value":"sunny"},{"value":"rainy"},{"value":"cloudy"}]}]}project_tag=ProjectTag(**tag)#should provided client_alias if calling from clientclient.add_project_tag(project_id=10,project_tag=project_tag,client_alias=client.alias)#ORproject.add_project_tag(project_tag=project_tag)Edit Project Tags** Note:Can not edit project tag that does not existCan not modify the data type of existing project tagsCan not provide attributes with existing optionstag={"attributes":[{"name":"weather","type":"option","options":[{"value":"unknown"},{"value":"snowy"}]}]}project_tag=ProjectTag(**tag)#should provided client_alias if calling from clientclient.edit_project_tag(project_id=10,project_tag=project_tag,client_alias=client.alias)#ORproject.edit_project_tag(project_tag=project_tag)Add New Ontology ClassesNote: Can not add existing ontology class!new_classes=[OntologyClass(name="obstruction",rank=9,color="#AB4321",attributes=[{"name":"status","type":"option","options":[{"value":"static"},{"value":"moving"}]}])]#should provided client_alias if calling from clientclient.add_ontology_classes(project_id=24,ontology_classes=new_classes,client_alias=client.alias)#ORproject.add_ontology_classes(ontology_classes=new_classes)Edit Ontology Classes** Note:Can not edit ontology class that does not existCan not modify the data type of existing ontology class attributesCan not provide attributes with existing optionsedit_classes=[OntologyClass(name="obstruction",color="#AB4321",attributes=[{"name":"status","type":"option","options":[{"value":"unknown"}]}])]#should provided client_alias if calling from clientclient.edit_ontology_classes(project_id=24,ontology_classes=edit_classes,client_alias=client.alias)#ORproject.edit_ontology_classes(ontology_classes=edit_classes)Create DatasetUsecreate_datasetto import dataset fromcloud storagedataset_data={"name":"Dataset 1","data_source":DataSource.Azure/DataSource.AWS,"storage_url":"storage/url","container_name":"azure container name","data_folder":"datafolder/to/vai_anno","sensors":project.sensors,"type":DatasetType.ANNOTATED_DATA,"annotation_format":AnnotationFormat.VISION_AI,"annotations":["groundtruth"],"sequential":False,"render_pcd":False,"generate_metadata":False,"auto_tagging":["timeofday"],"sas_token":"azure sas token",# only for azure storage"access_key_id":"aws s3 access key id",# only for private s3 bucket, don't need to assign it in case of public s3 bucket or azure data source"secret_access_key":"aws s3 secret access key"# only for private s3 bucket, don't need to assign it in case of public s3 bucket or azure data source}dataset=project.create_dataset(**dataset_data)Input arguments for creating dataset from cloud storage:Argument nameType/OptionsDefaultDescriptionnamestr＊--name of your datasetdata_sourceDataSource.AzureDataSource.AWS＊--the datasource of your datasetstorage_urlstr＊--your cloud storage urlcontainer_namestrNoneazure container namedata_folderstr＊--the relative data folder from the storage_url and containersensorslist[Sensor]＊--the list of Sensor of your dataset (one or more from project specified sensors)typeDatasetType.ANNOTATED_DATADatasetType.RAW_DATA＊--your dataset type  (annotated or raw data)annotation_formatAnnotationFormat.VISION_AIAnnotationFormat.KITTIAnnotationFormat.COCOAnnotationFormat.IMAGE＊--the format of your annotation dataannotationslist[str]Nonelist of names for your annotation data folders, such as ["groundtruth"]sequentialboolFalsedata is sequential or notrender_pcdboolFalserender pcd preview image or notgenerate_metadataboolFalsegenerate image meta data or notauto_tagginglistNonegenerate auto_tagging with target models["weather", "scene", "timeofday"]descriptionstrNoneyour dataset descriptionsas_tokenstrNoneSAStoken for azure containeraccess_key_idstrNoneaccess key id for AWS private s3 bucketsecret_access_keystrNonesecret access key for AWS private s3 bucket＊--: required argument without defaultCheckhttps://linkervision.gitbook.io/dataverse/data-management/import-datasetfor the detail ofImport Dataset.Get DatasetTheget_datasetmethod retrieves the dataset info from the connected site. Thedataset_idparameter is the unique integer ID of the dataset, not its "name" property.dataset=client.get_dataset(dataset_id=5)List ModelsThelist_modelsmethod will list all the models in the given project#1models=client.list_models(project_id=1,client_alias=client.alias)#2project=client.get_project(project_id=1)models=project.list_models()Get ModelTheget_modelmethod will get the model detail info by the given model-idmodel=client.get_model(model_id=30,client_alias=client.alias)model=project.get_model(model_id=30)From the given model, we could get the label file / triton model file / onnx model file by the commands below.status,label_file_path=model.get_label_file(save_path="./labels.txt",timeout=6000)status,triton_model_path=model.get_triton_model_file(save_path="./model.zip",timeout=6000)status,onnx_model_path=model.get_onnx_model_file(save_path="./model.onnx",timeout=6000)TroubleshootingNext stepsContributingLinks to language reposPython Readme