Package: prefect-fugue
Prefect Fugue IntegrationThis project provides the Fugue tasks, context and blocks for Prefect.Getting StartedFugue is a unified interface for distributed computing that lets users execute Python, pandas, and SQL code on Spark, Dask and Ray without rewrites.The most common use cases are:Accelerating or scaling existing Python and pandas code by bringing it to Spark or Dask with minimal rewrites.
Using FugueSQL to define end-to-end workflows on top of pandas, Spark, and Dask DataFrames. FugueSQL is an enhanced SQL interface that can invoke Python code with added keywords.Maintaining one codebase for pandas, Spark, Dask and Ray projects. Logic and execution are decoupled through Fugue, enabling users to be focused on their business logic rather than writing framework-specific code.Improving iteration speed of big data projects. Fugue seamlessly scales execution to big data after local development and testing. By removing PySpark code, unit tests can be written in Python or pandas and ran locally without spinning up a cluster.The best way to get started with Fugue is to work through the 10 minute tutorials:Fugue in 10 minutesFugueSQL in 10 minutesPython setupRequires an installation of Python 3.8+.We recommend using a Python virtual environment manager such as pipenv, conda or virtualenv.InstallationInstallprefect-fuguewithpip:pipinstallprefect-fugueIt's also recommended to register Fugue blocks into your current Prefect workspace:prefectblockregister-mprefect_fugueCreating a BlockThis will allow a creation of the Fugue Engine blockThere are 4 items that need to be filled to create a block.Block Name - name that will be used to use the block.Engine Name - one of the Fugue supported backends (spark, dask, ray, duckdb)Engine Config - configurations related to the clusterSecret Config - credentials to connect to a clusterFor example, a Databricks Block could look like:Block Name - databricksEngine Name - sparkEngine Config - NoneSecret Config - seen below{"host":"https://dbc-38aaa459-faaf.cloud.databricks.com","token":"dapiecaaae64a727498daaaaafe1bace968a","cluster_id":"0612-191111-6fopaaaa"}As long as you installedprefect_fugue, Fugue is able to recognize and convert a block expression to aFugueExecutionEngine. For example if you have a block with pathfugue/databricks, then the expressionprefect:fugue/databricksbecomes a valid execution engine expression. When fugue parse this expression, it will load theBlockfromfugue/databricks, then base on the fields of the block, it will construct aDatabricksExecutionEnginefor your Fugue operations.Using a Spark Cluster Inside a FlowLet’s start by running code on top of Databricks.databricks-connectis already installed in this environment. This section may have a lot of logs because of the monitoring provided by Prefect. This section also assumes that the user has Prefect configured to the right workspace.Below we have one task that takes in aSparkSessionand uses it to run some Spark code. We can then use this in the Prefect Flow with thefugue.api.engine_context. This will create an ephemeral cluster to run the code underneath, and then turn off when finished.fromprefectimporttask,flowimportfugue.apiasfa@taskdefmy_spark_task(spark,n=1):df=spark.createDataFrame([[f"hello spark{n}"]],"a string")df.show()@flowdefspark_flow(engine):withfa.engine_context(engine)asspark_engine:my_spark_task(spark_engine.spark_session,1)spark_flow("prefect:fugue/databricks")# pay attention to the engine expressionSimilarly, if you don’t use Databricks but have your own way to get aSparkSession, you can directly pass theSparkSessioninto the Flow.frompyspark.sqlimportSparkSessionspark=SparkSession.builder.getOrCreate()spark_flow(spark)More Flexibilityfugue.api.engine_contextcreates a context under which all fugue operations will use the context engine by default. This works either the fugue code is directly under the context or inside the tasks or flows that are invoked under this context.fromprefectimporttask,flowimportfugue.apiasfadefmy_transformer(df:pd.DataFrame)->pd.DataFrame:returndf@taskdefsub_task(path:str):df=fa.load(path)df=fa.transform(df,my_transformer,schema="*")df.save(path+".output.parquet")@flowdefsub_flow(path:str):df=fa.load(path)df=fa.transform(df,my_transformer,schema="*")df.save(path+".output.parquet")@flowdefmain_flow(path,engine=None):withfa.engine_context(engine)asspark_engine:sub_task(path)sub_flow(path)main_flow("<local path>")# test locally, all tasks and flows run without Sparkmain_flow("<dbfs path>","prefect:fugue/databricks")# all tasks and flows will run on DatabrickesTesting Locally Before Running Map Jobs on Spark, Dask, and RayWe showed how to run Spark code on top of a Spark cluster, but the strength of Fugue is decoupling from distributed framework code such as Spark and Dask. Decoupling from these frameworks allows us to test code locally before scaling out to a cluster. In the example below, we simulate having a pandas DataFrame where each row is a job.When testing the Flow, we can passNoneas the engine so everything runs on Pandas. When ready to scale out, we can pass in ourBlockorSparkSession. Fugue’stransform()task will use the engine provided by thefugue_enginecontext.fromtimeimportsleepimportpandasaspdimportfugue.apiasfafromprefectimporttask,flow@taskdefcreate_jobs(n)->pd.DataFrame:returnpd.DataFrame(dict(jobs=range(n)))# schema: *,batch_size:strdefrun_one_job(df:pd.DataFrame)->pd.DataFrame:sleep(len(df)*5)returndf.assign(batch_size=len(df))@flowdefrun_all_jobs(n,engine=None):jobs=create_jobs(n)withfa.engine_context(engine):returntransform(jobs,run_one_job,partition="per_row",as_local=True)We can test the Flow above on a local machine without Spark. We run on one job first.run_all_jobs(1)# run locally on PandasBecasue it succeeded, we can now attach our Fugue DatabricksBlockto run on Databricks. Now we run on 8 jobs, and we’ll see that parallelization from the Spark cluster will make this Flow execute faster.run_all_jobs(8,"prefect:fugue/databricks")# run on databricksThere is still some overhead with sending the work, but the time is decreased compared to the expected execution time if ran sequentially (40 seconds).We can also use local Dask by passing the string"dask". We can also pass aDask Client()or use the Fugue EngineBlockwithCoiled. More information can be found in theCoiled cloudprovider docs.run_all_jobs(4,dask_client)Running SQL on any Spark, Dask, and DuckdbPrototyping locally, and then running the full job on the cluster is also possible withFugueSQL. DuckDB is a good engine to run SQL queries on flat files or Pandas DataFrames. When ready, we can bring it to SparkSQL on the cluster. Similar to the transform() task shown above, there is also an fsql() task.Here we can load in data and perform a query with FugueSQL. FugueSQL has additional keywords such as LOAD and SAVE so we can run everything from loading, processing, and saving all on DuckDB or SparkSQL. More information on FugueSQL can be found in the FugueSQL tutorials.importfugue.apiasfa@flowdefrun_sql(top,engine):withfa.engine_context(engine):fa.fugue_sql_flow("""df = LOAD "https://d37ci6vzurychx.cloudfront.net/trip-data/green_tripdata_2022-01.parquet"SELECT PULocationID, COUNT(*) AS ct FROM dfGROUP BY 1 ORDER BY 2 DESC LIMIT {{top}}PRINT""",top=top).run()To debug locally without SparkSQL, we can use DuckDB as the engine.run_sql(2,"duckdb");# debug/develop without sparkAgain to run on the cluster, we can use the Databricks Block.run_sql(10,"prefect:fugue/databricks")ResourcesIf you encounter any bugs while usingprefect-fugue, feel free to open an issue in theprefect-fuguerepository.If you have any questions or issues while usingprefect-fugue, you can find help in theFugue Slack community.DevelopmentIf you'd like to install a version ofprefect-fuguefor development, clone the repository and perform an editable install withpip:gitclonehttps://github.com/fugue-project/prefect-fugue.gitcdprefect-fugue/

pipinstall-e".[dev]"# Install linting pre-commit hookspre-commitinstall