Package: hyperleaup
hyperleaupPronounced "hyper-loop". Create and publish Tableau Hyper files from Apache Spark DataFrames or Spark SQL.Why are data extracts areso slow?Tableau Data Extracts can take hours to create and publish to a Tableau Server.
Sometimes this means waiting around most of the day for the data extract to complete.
What a waste of time! In addition, the Tableau Backgrounder (the Tableau Server job scheduler)
becomes a single point of failure as more refresh jobs are scheduled and long running jobs exhaust the server’s resources.How hyperleaup helpsRather than pulling data from the source over an ODBC connection,hyperleaupcan write data directly to a Hyper file
and publish final Hyper files to a Tableau Server. Best of all, you can take advantage of all the benefits of
Apache Spark + Tableau Hyper API:perform efficient CDC upsertsdistributed read/write/transformations from multiple sourcesexecute SQL directlyhyperleaupallows you to create repeatable data extracts that can be scheduled to run on a repeated frequency
or even incorporate it as a final step in an ETL pipeline, e.g. refresh data extract with latest CDC.Getting StartedA list of usage examples is available in thedemofolder of this repo as aDatabricks Notebook Archive (DBC)or IPython Notebook (demo/Hyperleaup-Demo.ipynb).Example usageThe following code snippet creates a Tableau Hyper file from a Spark SQL statement and publishes it as a datasource to a Tableau Server.fromhyperleaupimportHyperFile# Step 1: Create a Hyper File from Spark SQLquery="""select *from transaction_historywhere action_date > '2015-01-01'"""hf=HyperFile(name="transaction_history",sql=query,is_dbfs_enabled=True)# Step 2: Publish Hyper File to a Tableau Serverhf.publish(tableau_server_url,username,password,site_name,project_name,datasource_name)# Step 3: Append new datanew_data="""select *from transaction_historywhere action_date > last_publish_date"""hf.append(sql=new_data)Creation ModeThere are several options for how to create the Hyper file that can be set by adding argumentcreation_modewhen initializing HyperFile instance. The default is PARQUET.ModeDescriptionData SizePARQUETSaves data to a single Parquet file then copies to Hyper file.MEDIUMCOPYSaves data to CSV format then copies to Hyper file.MEDIUMINSERTReads data into memory; more forgiving for null values.SMALLLARGEFILESaves data to multiple Parquet files then copies to Hyper file.LARGEExample of setting creation mode:hf = HyperFile(name="transaction_history", sql=query, is_dbfs_enabled=True, creation_mode="PARQUET")Hyper File OptionsThere is an optionalHyperFileConfigthat can be used to change default behaviors.timestamp_with_timezone:IfTrue, use timestamptz datatype with HyperFile. Recommended if using timestamp values with Parquet create mode. (default=False)allow_nulls:IfTrue, skip default behavior of replacing null numeric and strings with non-null values. (default=False)convert_decimal_precision:IfTrue, automatically convert decimals with precision over 18 down to 18. This has risk of data truncation. (default=False)Example using configsfromhyperleaupimportHyperFile,HyperFileConfighf_config=HyperFileConfig(timestamp_with_timezone=True,allow_nulls=False,convert_decimal_precision=False)hf=HyperFile(name="transaction_history",sql=query,is_dbfs_enabled=True)Legal InformationThis software is providedas-isand is not officially supported by Databricks through customer technical support channels.
Support, questions, and feature requests can be submitted through the Issues page of this repo.
Please understand that issues with the use of this code will not be answered or investigated by Databricks Support.Core Contribution teamLead Developer:Will Girten, Lead SSA @DatabricksPuru Shrestha, Sr. BI DeveloperProject SupportPlease note that all projects in the /databrickslabs github account are provided for your exploration only,
and are not formally supported by Databricks with Service Level Agreements (SLAs).They are provided AS-IS and we do not make any guarantees of any kind.Please do not submit a support ticket relating to any issues arising from the use of these projects.Any issues discovered through the use of this project should be filed as GitHub Issues on the Repo.They will be reviewed as time permits, but there are no formal SLAs for support.Building the ProjectTo build the project:python3 -m buildRunning PytestsTo run tests on the project:cd tests
python test_hyper_file.py
python test_creator.py