Package: dbldatagen
Databricks Labs Data Generator (dbldatagen)Project DescriptionThedbldatagenDatabricks Labs project is a Python library for generating synthetic data within the Databricks
environment using Spark. The generated data may be used for testing, benchmarking, demos, and many
other uses.It operates by defining a data generation specification in code that controls
how the synthetic data is generated.
The specification may incorporate the use of existing schemas or create data in an ad-hoc fashion.It has no dependencies on any libraries that are not already installed in the Databricks
runtime, and you can use it from Scala, R or other languages by defining
a view over the generated data.Feature SummaryIt supports:Generating synthetic data at scale up to billions of rows within minutes using appropriately sized clustersGenerating repeatable, predictable data supporting the need for producing multiple tables, Change Data Capture,
merge and join scenarios with consistency between primary and foreign keysGenerating synthetic data for all of the
Spark SQL supported primitive types as a Spark data frame which may be persisted,
saved to external storage or
used in other computationsGenerating ranges of dates, timestamps, and numeric valuesGeneration of discrete values - both numeric and textGeneration of values at random and based on the values of other fields
(either based on thehashof the underlying values or the values themselves)Ability to specify a distribution for random data generationGenerating arrays of values for ML-style feature arraysApplying weights to the occurrence of valuesGenerating values to conform to a schema or independent of an existing schemause of SQL expressions in synthetic data generationplugin mechanism to allow use of 3rd party libraries such as FakerUse within a Databricks Delta Live Tables pipeline as a synthetic data generation sourceGenerate synthetic data generation code from existing schema or data (experimental)Details of these features can be found in the online documentation  -online documentation.DocumentationPlease refer to theonline documentationfor
details of use and many examples.Release notes and details of the latest changes for this specific release
can be found in the GitHub repositoryhereInstallationUsepip install dbldatagento install the PyPi package.Within a Databricks notebook, invoke the following in a notebook cell%pip install dbldatagenThe Pip install command can be invoked within a Databricks notebook, a Delta Live Tables pipeline
and even works on the Databricks community edition.The documentationinstallation notescontains details of installation using alternative mechanisms.CompatibilityThe Databricks Labs Data Generator framework can be used with Pyspark 3.1.2 and Python 3.8 or later. These are
compatible with the Databricks runtime 10.4 LTS and later releases. For full Unity Catalog support,
we recommend using Databricks runtime 13.2 or later (Databricks 13.3 LTS or above preferred)For full library compatibility for a specific Databricks Spark release, see the Databricks
release notes for library compatibilityhttps://docs.databricks.com/release-notes/runtime/releases.htmlWhen using the Databricks Labs Data Generator on "Unity Catalog" enabled Databricks environments,
the Data Generator requires the use ofSingle UserorNo Isolation Sharedaccess modes when using Databricks
runtimes prior to release 13.2. This is because some needed features are not available inSharedmode (for example, use of 3rd party libraries, use of Python UDFs) in these releases.
Depending on settings, theCustomaccess mode may be supported.The use of Unity CatalogSharedaccess mode is supported in Databricks runtimes from Databricks runtime release 13.2
onwards.See the following documentation for more information:https://docs.databricks.com/data-governance/unity-catalog/compute.htmlUsing the Data GeneratorTo use the data generator, install the library using the%pip installmethod or install the Python wheel directly
in your environment.Once the library has been installed, you can use it to generate a data frame composed of synthetic data.For exampleimport dbldatagen as dg
from pyspark.sql.types import IntegerType, FloatType, StringType
column_count = 10
data_rows = 1000 * 1000
df_spec = (dg.DataGenerator(spark, name="test_data_set1", rows=data_rows,
                                                  partitions=4)
           .withIdOutput()
           .withColumn("r", FloatType(), 
                            expr="floor(rand() * 350) * (86400 + 3600)",
                            numColumns=column_count)
           .withColumn("code1", IntegerType(), minValue=100, maxValue=200)
           .withColumn("code2", IntegerType(), minValue=0, maxValue=10)
           .withColumn("code3", StringType(), values=['a', 'b', 'c'])
           .withColumn("code4", StringType(), values=['a', 'b', 'c'], 
                          random=True)
           .withColumn("code5", StringType(), values=['a', 'b', 'c'], 
                          random=True, weights=[9, 1, 1])
 
           )
                            
df = df_spec.build()
num_rows=df.count()Refer to theonline documentationfor further
examples.The GitHub repository also contains further examples in the examples directory.Spark and Databricks Runtime CompatibilityThedbldatagenpackage is intended to be compatible with recent LTS versions of the Databricks runtime, including
older LTS versions at least from 10.4 LTS and later. It also aims to be compatible with Delta Live Table runtimes,
includingcurrentandpreview.While we don't specifically drop support for older runtimes, changes in Pyspark APIs or
APIs from dependent packages such asnumpy,pandas,pyarrow, andpyparsingmake cause issues with older
runtimes.By design, installingdbldatagendoes not install releases of dependent packages in order
to preserve the curated set of packages pre-installed in any Databricks runtime environment.When building on local environments, the build process uses thePipfileand requirements files to determine
the package versions for releases and unit tests.Project SupportPlease note that all projects released underDatabricks Labsare provided for your exploration only, and are not formally supported by Databricks with Service Level Agreements
(SLAs).  They are provided AS-IS, and we do not make any guarantees of any kind.  Please do not submit a support ticket
relating to any issues arising from the use of these projects.Any issues discovered through the use of this project should be filed as issues on the GitHub Repo.They will be reviewed as time permits, but there are no formal SLAs for support.FeedbackIssues with the application?  Found a bug?  Have a great idea for an addition?
Feel free to file anissue.