Package: pyspark-extension
Spark ExtensionThis project provides extensions to theApache Spark projectin Scala and Python:Diff:Adifftransformation and application forDatasets that computes the differences between
two datasets, i.e. which rows toadd,deleteorchangeto get from one dataset to the other.Global Row Number:AwithRowNumberstransformation that provides the global row number w.r.t.
the current order of the Dataset, or any given order. In contrast to the existing SQL functionrow_number, which
requires a window spec, this transformation provides the row number across the entire Dataset without scaling problems.Inspect Parquet files:The structure of Parquet files (the metadata, not the data stored in Parquet) can be inspected similar toparquet-toolsorparquet-cliby reading from a simple Spark data source.
This simplifies identifying why some Parquet files cannot be split by Spark into scalable partitions..Net DateTime.Ticks:Convert .Net (C#, F#, Visual Basic)DateTime.Ticksinto Spark timestamps, seconds and nanoseconds.Available methods:// ScaladotNetTicksToTimestamp(Column):Column// returns timestamp as TimestampTypedotNetTicksToUnixEpoch(Column):Column// returns Unix epoch seconds as DecimalTypedotNetTicksToUnixEpochNanos(Column):Column// returns Unix epoch nanoseconds as LongTypeThe reverse is provided by (all returnLongType.Net ticks):// ScalatimestampToDotNetTicks(Column):ColumnunixEpochToDotNetTicks(Column):ColumnunixEpochNanosToDotNetTicks(Column):ColumnThese methods are also available in Python:# Pythondotnet_ticks_to_timestamp(column_or_name)# returns timestamp as TimestampTypedotnet_ticks_to_unix_epoch(column_or_name)# returns Unix epoch seconds as DecimalTypedotnet_ticks_to_unix_epoch_nanos(column_or_name)# returns Unix epoch nanoseconds as LongTypetimestamp_to_dotnet_ticks(column_or_name)unix_epoch_to_dotnet_ticks(column_or_name)unix_epoch_nanos_to_dotnet_ticks(column_or_name)Spark job description:Set Spark job description for all Spark jobs within a context:fromgresearch.sparkimportjob_description,append_job_descriptionwithjob_description("parquet file"):df=spark.read.parquet("data.parquet")withappend_job_description("count"):count=df.countwithappend_job_description("write"):df.write.csv("data.csv")For details, see theREADME.mdat the project homepage.Using Spark ExtensionPyPi package (local Spark cluster only)You may want to install thepyspark-extensionpython package from PyPi into your development environment.
This provides you code completion, typing and test capabilities during your development phase.Running your Python application on a Spark cluster will still require one of the ways below
to add the Scala package to the Spark environment.pipinstallpyspark-extension==2.11.0.3.4Note: Pick the right Spark version (here 3.4) depending on your PySpark version.PySpark APIStart a PySpark session with the Spark Extension dependency (version ≥1.1.0) as follows:frompyspark.sqlimportSparkSessionspark=SparkSession\.builder\.config("spark.jars.packages","uk.co.gresearch.spark:spark-extension_2.12:2.11.0-3.4")\.getOrCreate()Note: Pick the right Scala version (here 2.12) and Spark version (here 3.4) depending on your PySpark version.PySpark REPLLaunch the Python Spark REPL with the Spark Extension dependency (version ≥1.1.0) as follows:pyspark--packagesuk.co.gresearch.spark:spark-extension_2.12:2.11.0-3.4Note: Pick the right Scala version (here 2.12) and Spark version (here 3.4) depending on your PySpark version.PySparkspark-submitRun your Python scripts that use PySpark viaspark-submit:spark-submit--packagesuk.co.gresearch.spark:spark-extension_2.12:2.11.0-3.4[script.py]Note: Pick the right Scala version (here 2.12) and Spark version (here 3.4) depending on your Spark version.Your favorite Data Science notebookThere are plenty ofData Science notebooksaround. To use this library,
adda jar dependencyto your notebook using theseMaven coordinates:uk.co.gresearch.spark:spark-extension_2.12:2.11.0-3.4Ordownload the jarand place it
on a filesystem where it is accessible by the notebook, and reference that jar file directly.Check the documentation of your favorite notebook to learn how to add jars to your Spark environment.