Package: hdlshare
Prepare UsageCreate CertificatesSetup a PKI-Certificate Server (Preferably in the same subaccount but it is not a requirement)Download Service-keyRun sapcert-cmdsapcert-pservice_keys/xpm-hdlfs-ds.json-dcertificates-v60-tDAYS<CN>Creates 2 files.key - private key.pem - certificate chainCreate HDLFS InstanceUsing theService Manager API.The subaccount needs a runningService Managerin the subaccount.For creating and managing HDL instances I have developed a commandline toolhldinst.Using default service-key file: service_keys/sm-sk.txtList instanceshdlinstancelistAccess Parameterhdlinstparamsxpm-mt1Instance Detailshdlinstdetailsxpm-mt1Currently the hdlfs-endpoint is not provided by the hdl-instance details but must be build by.files.hdl., e.g.40e60329-e854-46e2-89d4-728093fb7576.files.hdl.prod-us30.hanacloud.ondemand.comThe following command adds a config section to the configuration file: $HOME/.hdlfscli.config.jsonhdlinstadd2configxpm-mt1-Chdlmt1To create a new hdl-instancehdlinstcreatexpm-hdl-mt1--CommonNamehdlmt1-cThis uses a template file in folder'./configs/hdlfs_template.json'. The argument '-c' is needed for using the subject of the certificate with the name <certificates-folder><CommonName>.pem as the user subject for the HDLFs authentication.Additional Requirements for HDL Instance managementAdding API for deleting an HDLFS instanceAdding an API for managing the user/parameter of HDL instance. Important when havin many hdl-instances to adminster.Delta Sharing with SAP HDLResources:Git DocumentationSwagger for hdlfs-serviceOverviewThe usage of Spark is required for big data that needs a cluster of compute nodes and cheap storage (delta lake). The result of processing big data needs to be HANA tables that can be consumed by DataSphere. In order to share the results with other applications you either need asystem - to - systemintegration where the credentials of the target applications are shared or"data product"-kind of integration where you expose the data to external users with an external access management like "delta sharing".Delta Sharing is internally used as a data sharing technology that separates the internals of the data producer from the data consumer and adds further options to govern the access. For eachHANA Cloud, Data Lakeinstance a Delta Sharing server can be activated. For the time being this is only enabled for internal use. Once we have thoroughly tested this API by productive usage within SAP and if this service can be offered to the main Hyperscaler we might consider opening this service for SAP customer.URLsCatalog API.files.hdl./catalog/v2
.files.hdl./policies/v2Audience in JWT.files.Delta SharingToken Access: .sharing.hdl./shares/v1/Cert Access: .files.hdl./shares/v1/Overview HDL ToolsCurrently the only way to manageDelta Sharingare mainly using RestAPIs. For testing purpose how to best use theDelta Sharingmanagement I have developed a command line tools.hdlfscliFor manageing the files on HDLfs the Command line toolhdlfscliused. This can be downloaded from gitbigdataservices/hdlfs-cli.hdlfscli-h
hdlfsclimanagesinteractionwithHDLFilescloudservice.

Findmoreinformationat:https://help.sap.com/docs/hana-cloud-data-lake/client-interfaces/hdlfscli-data-lake-files-utility

StorageCommands(CloudStorage):lsorlistListafileordirectorylsrortreeListafileordirectorywithrecursivetraversaluploadUploadafileordirectorytoremotefilestoragefromlocalfilestorage,creatingremotedirectoriessuitablyrmordeleteDeleteafileordirectory.Usewiththeflag'-f'fordirectoriese.g.,'rm -f <directory>'todeleteadirectorymvMovearemotefilefromremotesourcepathtoremotedestinationpathcatOpenafiledownloadCopyafileordirectoryfromremotestoragetolocalfilestorage

JWTCommands(ManageJSONWebTokens):jwtManageJWT

UsageForStorageCommands:hdlfscli[storage-options]storage-command[arguments]UsageForJWTCommands:hdlfsclijwt[jwt-command][jwt-options]Use"hdlfscli storage-options"foralistofglobalcommand-lineoptions(appliestoallstoragecommands).
Use"hdlfscli jwt-options"foralistofjwtcommand-lineoptions(appliestoalljwtcommands).Thishdlfscliuses a configuration file at $HOME/.hdlfscli.config.json. This config is also been used for the commandline apps"hdlshare"and"hdlpolicy".For convenience reasons I have created a'default' - config-section in .hdlfscli.config.json. This is used when no config is given but requiredalias with hdl='hdlfscli -config'Example%hdldefaultlsdata/deltalake
DIRECTORYnobodynogroup00FR
DIRECTORYnobodynogroup00DE
DIRECTORYnobodynogroup00US
%hdldefaultuploaddata/US/customerdata/deltalake/US/customerSetup HDLsapcert - create signed certificateshdlinstance - create a HDLinstance using the created instanceHDL Delta Sharing managementhdlshare - manage HDL Delta Shareshdlpolicy - manage HDL Delta Share policysdsclient - Delta Sharing Client to test HDL Delta SharingInstallationClone the git repositorypython -m buildpip install .Install via pip (not yet)pip install hdlshareHDL ShareCreates and manages shares. The command line app is using the RestAPI (Swagger for hdlfs-service)delta_sharing_python_client%hdlshare-h
usage:ManageHDLFSshares[-h][-r][-m][-C][-pPATH][-cCONFIG]{list,add,delete,get}[target...]positionalarguments:{list,add,delete,get}Commandfor'target'-argumenttargetshareschematable(optional)options:-h,--helpshowthishelpmessageandexit-r,--recursiveListrecursively-m,--metadataShowmetadataoftable(action=list)-C,--cascadeDropcascadewhendeletingshare(action=delete)-pPATH,--pathPATHHDLFSdatafolder-cCONFIG,--configCONFIGHDLFsconfigin'.hdlfscli.config.json'ExamplesList data on hdlfs:%hdldefaultlsdata/deltalake/USDIRECTORYnobodynogroup00personsList all shares and tables%hdlsharelist-r

shares
├──sbm
├──hxm
└──crm└──us└──customerAdd new table to share%hdlshareaddhxmusemployees--pathdata/deltalake/persons
Tablesuccessfullyadded:hxm:us.employees
%hdlsharelist-r
shares
├──sbm
├──hxm
│└──us
│└──employees
└──crm└──us└──customerDetails of Share:schema:table:%hdlsharelist-rm
shares
├──sbm
├──hxm
│└──us
│└──employees
│├──data/deltalake/persons
│├──DELTA
│└──cdf:True
└──crm└──us└──customer├──data/deltalake/US/customer├──DELTA└──cdf:TruehdlpolicyCreate and manage policies.Documentationswagger%hdlpolicy-h
usage:ManageHDLFSsharepolicies[-h][-pPOLICY][-sSUBJECT][-RRESOURCE][-PPRIVILEGE][-CCONSTRAINT][-DDAYS][-cCONFIG]{list,add,delete,copy,token,showtoken}[policy_names...]positionalarguments:{list,add,delete,copy,token,showtoken}Actionpolicy_namesPolicyname(for'copy'arg2policies)options:-h,--helpshowthishelpmessageandexit-pPOLICY,--policyPOLICYPolicycontent(json)-sSUBJECT,--subjectSUBJECTsubject/usertoaddordeletefrompolicyandforshowingorgeneratingtokens-RRESOURCE,--resourceRESOURCEResourcetoaddordeletefrompolicy-PPRIVILEGE,--privilegePRIVILEGEPrivilegetoaddordeletefrompolicy-CCONSTRAINT,--constraintCONSTRAINTConstrainttoaddordeletefrompolicy-DDAYS,--daysDAYSDaysbeforeexpiringfromnowon.-cCONFIG,--configCONFIGHDLFsconfigin'.hdlfscli.config.json'ExamplesList all policies%hdlpolicylistCopy policy%hdlpolicycopyde_regionnl_regionAdd resource to policy%hdlpolicyaddnl_region-Rshare:crmDelete subject/user to policy%hdlpolicydeletenl_region-suser:de_adminAdd subject/user to policy%hdlpolicyaddnl_region-suser:hr_nlCreate token for user%hdlpolicytoken-shr_nlhdlclient - Delta Sharing Clienthdlclient-h
usage:hdlclient[-h][-r][-pPATH][-m][-vVERSION][-eEND_VERSION][-cCONFIG][-H][-s]profile{list,download,metadata}[target...]positionalarguments:profileProfileofdeltasharing{list,download,metadata}Actiontarget(optional)Target:<share>[<schema>][<table>]].

options:-h,--helpshowthishelpmessageandexit-r,--recursiveSyncfileswithhana-pPATH,--pathPATHDirectorytostoredata.-m,--metaDownloadmetadataascsn-filetoeditbeforestartingthereplication.-vVERSION,--versionVERSIONStartversion(Warning:overruledbymetadatastoredversion)-eEND_VERSION,--end_versionEND_VERSIONVersionend-cCONFIG,--configCONFIGConfig-fileforHANAaccess(yamlwithurl,user,pwd,port)-H,--HanaUploadtohana-s,--syncSyncfileswithhanaExamplesList avaliable shares%hdlclientadminlist-r

shares
├──sbm
├──hxm
│└──us
│└──employees
└──crm└──us└──customer

%hdlclienthxm_mdlist

shares
└──hxm└──us└──employees