Package: azdsdr
An all-in-one data reader and tools in Python - AZDSDRAn all-in-one data reader and tools in Python - AZDSDRInstallationPotential installation errors and solutionsUse Kusto ReaderAzure CLI AuthenticationRun any Kusto queryShow Kusto tablesCreate an empty Kusto table from a CSV fileUpload data to KustoUse Dremio ReaderStep 1. Install Dremio ConnectorStep 2. Generate a Personal Access Token(PAT)Step 3. Configure driverDremio Sample QueryMove data with functions fromPipelinesclassExport Kusto data to local csv fileMove Dremio data to KustoData Toolsdisplay_allDisplay all dataframe rowsThanksUpdate LogsJan 24, 2024Jan 23, 2023Jan 18, 2023Jan 17, 2023Jan 10, 2023Dec 16, 2022Dec 10, 2022Dec 6, 2022This package includes data reader for DS to access data in a easy way.Covered data platforms:KustoAzure Blob Storage (Samples coming soon)DremioMicrosoft Cosmos - Not Azure Cosmos DB, the Microsoft Cosmos using Scope, now AKA Azure Data Lake (Samples coming soon)May cover in the future:Databricks/SparkMicrosoft SynapseDelta LakePostgresqlMicrosoft SQL ServerSQLiteBesides, the package also include functions fromPipelinesclass to help move data around:Dremio to KustoKusto to CSV fileInstallationThe module is test and usable for Python 3.10 and Python 3.9. Other versions(Python 3.6+) should also works.Use pip to install the package and all of the dependencespip install -U azdsdrThe-Uwill help update your old version to the newestOr, you can clone the repository and copy over thereaders.pyfile to your project folder.The installation will also install all the dependance packages automatrically.pandaspyodbcazure-cliazure-kusto-dataazure-kusto-ingestazure-storage-blobmatplotlibipythonipykernelIf you are working on a new build OS, the all-in-one installation will also save you time from installing individual packages one by one.Potential installation errors and solutionsMost of the time, all dependent packages should be successfully installed without any additional interfere. But you may still see error message based on different OS and Python version.Need elevated permissionError message:Error: Could not install packages due to an OSError: [Erron 13] Permission denied:...Solution:Start a new Windows terminal window with Administrator permission (Right click icon, and then "Run as administrator")Fail to installpyodbcUsually occurs in Linux and MacOS.Error messageBuilding wheel for pyodbc (setup.py) ... errorSolutionLinux: run this firstsudoapt-getinstallunixodbc-devhttps://github.com/mkleehammer/pyodbc/issues/276Macos: run this firstbrewinstallunixodbcexportLDFLAGS="-L/opt/homebrew/Cellar/unixodbc/2.3.9/lib"exportCPPFLAGS="-I/opt/homebrew/Cellar/unixodbc/2.3.9/include"Use Kusto ReaderAzure CLI AuthenticationBefore running the kusto query, please useaz loginTo login into Azure using AAD authentication. An authentication refresh token is generated by Azure and stored in your local machine. This token will be revoked after90 days of inactivity.For More details, readSign in with Azure CLI.After successufuly authenticated with AAD, you should be able to run the following code without any pop up auth request. The Kusto Reader is test in Windows 10, also works in Linux and Mac.Run any Kusto queryfromazdsdr.readersimportKustoReadercluster="https://help.kusto.windows.net"db="Samples"kr=KustoReader(cluster=cluster,db=db)kql="StormEvents | take 10"r=kr.run_kql(kql)The functionrun_kqlwill return a Pandas Dataframe object hold byr. Thekrobject will be reused in the following samples.Userun_kql_allto output multiple result set.kql='''StormEvents| take 10;StormEvents| summarize count()'''rs=kr.run_kql_all(kql=kql)forrinrs:display(r)Show Kusto tablesList all tables:kr.list_tables()List tables with folder keyword:kr.list_tables(folder_name='Covid19')Create an empty Kusto table from a CSV fileThis function can be used before uploading CSV data to Kusto table. Instead of manually creating a Kusto table from CSV schema, use this function to create a empty Kusto table based on CSV file automatically.Besides, you can also specify the table's folder name.kusto_table_name='target_kusto_table'folder_name='target_kusto_folder'csv_file_name='local_csv_path'kr.create_table_from_csv(kusto_table_name=kusto_table_name,csv_file_path=csv_file_name,kusto_folder=folder_name)Upload data to KustoBefore uploading your data to Kusto, please make sure you have the right table created to hold the data. Ideally, you can use the abovecreate_table_from_csvto create an empty table for you.To enable the data ingestion(upload), you should also initialize the KustoReader object with an additionalingest_cluster_strparameter. Here is a sample, you should ask your admin or doc to find out the ingestion cluster url.cluster="https://help.kusto.windows.net"ingest_cluster="https://help-ingest.kusto.windows.net"db="Samples"kr=KustoReader(cluster=cluster,db=db,ingest_cluster_str=ingest_cluster)Note that you will need to create a empty table with aligned table schema to hold the data.You can also save the dataframe objectyour_df_dataas CSV file first, and create a empty table from the csv file.your_df_data.to_csv('temp.csv',index=False)target_kusto_table='upload_df_to_kusto_test'kr.create_table_from_csv(kusto_table_name=target_kusto_table,kusto_folder='test',csv_file_path='temp.csv')print('create empty table done')Then upload Pandas Dataframe to Kusto:target_kusto_table='kusto_table_name'df_data=your_df_datakr.upload_df_to_kusto(target_table_name=target_kusto_table,df_data=df_data)kr.check_table_data(target_table_name=target_kusto_table)Upload CSV file to Kusto:target_kusto_table='kusto_table_name'csv_path='csv_file.csv'kr.upload_csv_to_kusto(target_table_name=target_kusto_table,csv_path=csv_path)Upload Azure Blob CSV file to Kusto, this is the best and fast way to upload massive csv data to Kusto table.target_kusto_table='kusto_table_name'blob_sas_url='the sas url you generate from Azure portal or Azure Storage Explorer, or azdsdr'kr.upload_csv_from_blob(target_table_name=kusto_table_name,blob_sas_url=blob_sas_url)I will cover how to generateblob_sas_urlin the Azure Blob Reader section. [TODO]Use Dremio ReaderStep 1. Install Dremio ConnectorYou will need to install the Dremio ODBC driver first to useDremioReaderfrom this package.For Windows userPlease download thedremio-connectorfile from the drivers folder.Step 2. Generate aPersonal Access Token(PAT)Recommend storing this personal access token in a safe location, such as a user environment variable on your local machine.Start Menu -> “Edit Environment variables For Your Account”.Click “New” under environment variables.Enter a new variable with name “DREMIO_TOKEN” and set the value to the PAT you generated earlier.Note: you will have to log out your Windows account and log in again to take the new env variable take effort.Step 3. Configure driverGo to Start Menu -> “ODBC Data Sources (64-bit)”.Under User DSN, click “Add”.Add Dremio Connector.Configure as follows:setData Source NameasDremio Connector.with your ownuser@example.comas the username.Do remember to replace the dremio host with your own host string.Click Ok/SaveFor Linux and Mac UserYou can download the driver fromDremio's ODBC Driverpage. It should be working in theory, haven't been test yet.Dremio Sample Queryfromazdsdr.readersimportDremioReaderimportosusername="name@host.com"#token       = "token string"token=os.environ.get("DREMIO_TOKEN")dr=DremioReader(username=username,token=token)sql='''select*from[workspace].[folder].[tablename]limit 10'''r=dr.run_sql(sql)Move data with functions fromPipelinesclassExport Kusto data to local csv file[TODO]When the export data is very large like exceed 1 billion rows, kusto will export data to several csv files. this function will automatically combine the data to one single CSV file in destination folder.Move Dremio data to Kusto[TODO]Data Toolsdisplay_allDisplay all dataframe rowsThe IPython'sdisplaycan display only limited rows of data. This tool can displayallorspecified rowsof data.fromazdsdr.toolsimportpd_toolsdisplay_all=pd_tools().display_all#...prepare pd data# display alldisplay_all(pd_data)# display top 20 rowsdisplay_all(pd_data,top=20)ThanksThe Dremio ODBC Reader solution is origin fromKC Munnings. Glory and credits belong to KC.Update LogsJan 24, 2024Addbar1_chartinvis_tools, so that you can plot bar chart usingvis_toolsclass.Jan 23, 2023AddGridandXY Axes Lableoption for 1 line and 2 lines chart.Jan 18, 2023Add walk-around and solutions to potential installation errors.Jan 17, 2023Addshow_data_labeloption forvis_tools'sline1_chartfunction.
If specify theshow_data_label=True, the chart will show each data point's value.Jan 10, 2023Add guid to the temp cosmos script file and temp middle stream file to avoid temp files collision.Dec 16, 2022Add functionget_table_schemaforKustoReaderAdd functionget_table_folderforKustoReaderDec 10, 2022Update Dremio Reader configuration document and screenshot.Dec 6, 2022Add functiondownload_file_listof classAzureBlobReaderto download a list of CSV file with the same schema and merge to one target CSV file.Add functiondelete_blob_filesof classAzureBlobReaderto delete a list of blob files.Addusage sample code.