Package: dataverse-api
dataverse-apiThedataverse-apipackage is an abstraction layer developed for allowing simple interaction with Microsoft Dataverse Web API.Table of ContentsDescriptionGetting StartedDevelopment environmentCode requirementsTestingTo doUsageDataverseClientInitialize DataverseClientCreate new EntityUpdate existing EntityDataverseEntityInitialize Entity interfaceReadCreateUpsertDeleteAdd and remove AttributesAdd and remove Alternate KeysDescriptionThe main goal of this project was to enable some use-cases against Dataverse that I wanted to explore for a work assignment, while getting some experience in programming and testing out different ways of setting up the codebase.The functionality I have built into this Dataverse wrapper constitutes the functionality I have wanted to explore myself.Most important is to enable creating, upserting, updating and deleting rows of data into Dataverse tables using common data structures, and implementing choices on how these requests are to be formed. For example, when creating new rows, the user can choose between individualPOSTrequests per row, combining data into batch requests against the$batchendpoint, or even to use theCreateMultipleDataverse action.The framework is written in Python 3.11, seeing as this runtime is available in the current release of Azure Functions.Getting startedUsage is fairly simple - authentication must be handled by the user. TheDataverseClientsimply accepts an already authorizedrequests.Sessionwith which to handle API requests.I suggest usingmsalandmsal-requests-authfor authenticating theSession. The examples below include this way of implementing auth:importosfrommsalimportConfidentialClientApplicationfrommsal_requests_auth.authimportClientCredentialAuthfromrequestsimportSessionfromdataverse_apiimportDataverseClient# Prepare Authapp_reg=ConfidentialClientApplication(client_id=os.getenv("app_id"),client_credential=os.getenv("client_secret"),authority=os.getenv("authority_url"),)environment_url=os.getenv("environment_url")auth=ClientCredentialAuth(client=app_reg,scopes=[environment_url+"/.default"])# Prepare Sessionsession=Session()session.auth=auth# Instantiate DataverseClientclient=DataverseClient(session=session,environment_url=environment_url)# Instantiate interface to Entityentity=client.entity(logical_name="organization")# Read data!entity.read(select=["name"])Development environmentpoetryis used for managing dependencies. To developdataverse-api, follow the below steps to set up your local environment:Install poetryif you haven't already.Clone repository:$ git clone git@github.com:MarcusRisanger/dataverse-api.gitMove into the newly created local repository:$ cd dataverse-apiCreate virtual environment and install dependencies:$ poetry installCode requirementsAll code must passruffstyle checks to be merged. It is recommended to install pre-commit hooks to ensure this locally before commiting code:$ poetry run pre-commit installEach public method, class and module should have docstrings. Docstrings are written in the Numpy style.TestingTo produce Coverage reports, run the following commands:$ poetry run coverage run -m pytest
$ poetry run coverage xmlTo Do:Documentation ..MetadataChoiceMultichoiceMoneyEntity implementationIllegal file extensionsUpload fileUpload imageUpload large fileAdd / remove relationshipsAdd relationships (single/collection valued) as attr on entityAdd columns as attr on entityAdd Tests:Add columnRemove columnAdd alternate keyRemove alternate keyImplementCreateMultiplerequest if creating more than 100 elements?https://learn.microsoft.com/en-us/power-apps/developer/data-platform/bulk-operations?tabs=webapi#createmultipleImplement BulkDelete Action?https://learn.microsoft.com/en-us/power-apps/developer/data-platform/delete-data-bulk?tabs=sdkUsageDataverseClientInitialize DataverseClientFor now, I've coded the framework around therequestslibrary, for good and bad! In the future, I will consider generalizing further, letting the user pass an authenticated requests handler of choice to the framework by specifying aProtocolto follow instead.To instantiate, pass arequests.Sessiontogether with a Dataverse environment URL to theDataverseClientconstructor:session=Session()session.auth=authenvironment_url=os.getenv("dataverse_url")client=DataverseClient(session=session,environment_url=environment_url)Create new EntityIt is possible to create a new Entity using theDataverseClient. This requires a fullEntityMetadatadefinition according to Dataverse standards. You can make this yourself and follow theMetadataDumperprotocol, or use the provideddefine_entityfunction.Thedefine_labelfunction makes it simple to generateLabelmetadata with correctLocalizedLabelsin its payload.In the example below, the optionalreturn_representationargument has been set toTrueto receive the full Entity metadata definition as created by Dataverse as part of the server response. The response can be parsed byEntityMetadataclassmethod to get a full fledged object for editing.fromdataverse_api.metadata.attributesimportStringAttributeMetadatafromdataverse_api.metadata.entityimportEntityMetadata,define_entityfromdataverse_api.utils.labelsimportdefine_labelnew_entity=define_entity(schema_name="new_name",attributes=[StringAttributeMetadata(schema_name="new_primary_col",is_primary_name=True,description=define_label("Primary column for Entity."),display_name=define_label("Autonumber Column"),auto_number_format="{SEQNUM:6}-#-{RANDSTRING:3}")],description=define_label("Entity Created by Client"),display_name=define_label("Programmatically Created Table"))resp=client.create_entity(new_entity,return_representation=True)entity_meta=EntityMetadata.model_validate_dataverse(resp.json())Update existing EntityYou can update an existing Entity definition easily by retrieving the Entity metadata definition, and reupload an adjusted version.Below is a simple example. Note that this method also supportsreturn_representationin the same manner as theDataverseClient.create_entity()method, if you want to return the edited Entity metadata as persisted in Dataverse.metadata=client.get_entity_definition("new_name")metadata.display_name.localized_labels[0].label="Overridden Display Name"client.update_entity(metadata)DataverseEntityInitialize interface with EntityTo initializes an interface with a specific Dataverse Entity, use theDataverseClient.entity()method. It returns aDataverseEntityobject that allows interaction with this specific entity.entity=client.entity(logical_name="foo")As of now, onlyLogicalNameis supported for instantiating a newDataverseEntityobject.ReadTheDataverseEntity.read()method has been furnished with the necessary arguments to do querying as specified in theMicrosoft Dataverse documentation.A simple example:data=entity.read(select=["name","address"],filter="salary gt 10000",top=5,order_by="salary desc")CreateTo create rows, you can use apandas.DataFrameor a simple construct like a list of dicts, where each dict contains the data for a single row.Below is an example of creating rows in the Entity by passing a dataframe and specifying that the creation method should be theCreateMultipleweb API Action. Thereturn_createdargument can be set toTrueif you need the IDs as reference.entity.create(data=df,mode="multiple",return_created=True)Note that the different modes provide different content whenreturn_createdis set toTrue- the script simply sets aPreferheader to include created data in the server response.For now the user may choose how to handle this based on the list ofrequests.Responseobjects that will be returned by the method.UpsertUpserting data into Dataverse is simple. If you are just updating existing data you may have the URI (Primary Attribute ID) in your data. You can then omit thealternate_keyargument.entity.upsert(data=df,alternate_key="my_key",mode="batch")DeleteTBDAdd and remove AttributesTBDAdd and remove Alternate KeysTBD