Package: hdsclient
General Use Case for Spark/DatabricksThe usage of Spark is required for big data that needs a cluster of compute nodes and cheap storage (delta lake). The result of processing big data needs to be HANA tables that can be consumed by DataSphere. In order to share the results with other applications you either need asystem - to - systemintegration where the credentials of the target applications are shared or"data product"-kind of integration where you expose the data to external users with an external access management like "delta sharing".The following client was developed to use the API of the Databricks delta-sharing-server of the Unity catalog.The data can be downloaded to a csv-file with the name<share><schema><table>.csvand uploded to a HANA Database. Because the standard spark catalog is based on the delta lake format there is no feature for defining primary keys. The reason might be that there are not inherent tests on a "primary key violation" when a new record is written. A "primary key" might therefore lead to false assumptions.For supportingprimary keysand DB-specific data types a csn-file can be used when a table is created in HANA. The csn-file needs to have the same name as the table file-name but with the ".csn"-extension.Used files:filename of table: <share><schema><table>.csvmetadata of table: <share><schema><table>_meta.jsondelta records of tables when CDF-enabled: <share><schema><table>_delta.csvCSN table definition: <share><schema><table>_delta.csnLast downloaded version: <share><schema><table>_version.csnLast uploaded hana version: <share><schema><table>_hana.csnAll files are stored in the CWD or to thepathgiven in the command options.CSN File CreationTo support the creation of a csn-file you can usepipinstallpycsn

pyscn-h
pycsn<csv-file>-p[<primarykeys>]-n[<tablenames>]usage:pycsn[-h][-oOUTPUT][-pPRIMARY_KEYS[PRIMARY_KEYS...]][-nNAMES[NAMES...]][-s][-bBUFFER]filenames[filenames...]Createscsn-filefrompandasDataFrame.

positionalarguments:filenamesDataFilenames(csv)options:-h,--helpshowthishelpmessageandexit-oOUTPUT,--outputOUTPUTOverwritedefaultfilename-pPRIMARY_KEYS[PRIMARY_KEYS...],--primary_keysPRIMARY_KEYS[PRIMARY_KEYS...]Addprimarykeys(Attentionforalltables!).-nNAMES[NAMES...],--namesNAMES[NAMES...]Settablenames.-s,--sqlCreatetablesql.-bBUFFER,--bufferBUFFERAdditionalstringbufferInstallationpip install dsclientDelta Sharing Commandline ClientLists and downloads files from Delta Sharing.usage: dsclient.py [-h] [-p PATH] [-d] [-s STARTING] [-e ENDING] [-m] [-c CONFIG_FILE] [-H] [-S] profile [table]

positional arguments:
  profile               Profile of delta sharing
  table                 (optional) Table URL: <share>.<schema>.<table>. If not given table can be selected from available table list.

options:
  -h, --help            show this help message and exit
  -p PATH, --path PATH  Directory to store data.
  -d, --delta           Capture delta feeds
  -s STARTING, --starting STARTING
                        Start version (int, min=1)m or timestamp (str,"2019-09-26T07:58:30.996+0200")
  -e ENDING, --ending ENDING
                        End version (int, min=1) or timestamp (str,"2019-09-26T07:58:30.996+0200")
  -m, --meta            Show metadata
  -c CONFIG_FILE, --config-file CONFIG_FILE
                        Config-file for HANA access (yaml with url, user, pwd, port)
  -H, --Hana            Upload to hana
  -S, --Sync            Sync files with hana