Package: airflow-provider-ray
Apache Airflow Provider for RayA provider you can install into your Airflow environment to access custom Ray XCom backends, Ray Hooks, and Ray Operators.🧪 Experimental VersionThis provider is an experimental alpha containing necessary components to
orchestrate and schedule Ray tasks using Airflow. It is actively maintained
and being developed to bring production-ready workflows to Ray using Airflow.
Thie release contains everything needed to begin building these workflows using
the Airlfow taskflow API.Current Release:0.2.1RequirementsVisit theRay Project pagefor more info on Ray.⚠️ The server version and client version (build) of Ray MUST be
the same.-Python Version >= 3.7-Airflow Version >= 2.0.0-Ray Version == 1.3.0-Filelock >= 3.0.0ModulesRay XCom Backend: Custom XCom backend
to assist operators in moving data between tasks using the Ray API with its
internal Plasma store, thereby allowing for in-memory distributed processing
and handling of large data objects.Ray Hook: Extension ofHttphook
that uses the Ray client to provide connections to the Ray Server.Ray Decorator: Task decorator
to be used with the task flow API, combining wrapping the existing airflow@taskdecorate withray.remotefunctionality, thereby executing each
task on the ray cluster.Configuration and UsageAdd the provider package wheel file to the root directory of your Airflow project.In your AirflowDockerfile, you will need to add an environment variable to
specify your custom backend, along with the provider wheel install. Add the following:FROMquay.io/astronomer/ap-airflow:2.0.2-1-buster-onbuildUSERrootRUNpipuninstallastronomer-airflow-version-check-yUSERastroENVAIRFLOW__CORE__XCOM_BACKEND=ray_provider.xcom.ray_backend.RayBackendCheck ap-airflow version, if unsure, change toap-airflow:latest-onbuildWe are using a Ray1.3.0and python version3.7. To get a bleeding edge
version of Ray, you can to follow this format to build the wheel url in yourrequirements.txtfile:pipinstallairflow-provider-rayConfigure Ray Locally. To run ray locally, you'll need a minimum 6GB of free
memory.To start, in your environment with ray installed, run:(venv)$raystart--num-cpus=8--object-store-memory=7000000000--headIf you have extra resources, you can bump the memory up.You should now be able to open the ray dashboard athttp://127.0.0.1:8265/.Start your Airflow environment and open the UI.In the Airflow UI, add anAirflow Poolwith the following:Pool(name):ray_worker_pool
Slots:25In the Airflow UI, add anAirflow Connectionwith the following:ConnId:ray_cluster_connection
ConnType:HTTP
Host:ClusterIPAddress,withbasicAuthparamsifneeded
Port:10001In your Airflow DAG python file, you must include the following in yourdefault_argsdictionary:fromray_provider.xcom.ray_backendimportRayBackend...default_args={'on_success_callback':RayBackend.on_success_callback,'on_failure_callback':RayBackend.on_failure_callback,...}@dag(default_args=default_args,..)defray_example_dag():# do stuffUsing the taskflow API, your airflow task should now use the@ray_taskdecorator for any ray task and add theray_conn_id,
parameter astask_args, like:fromray_provider.decoratorsimportray_taskdefault_args={'on_success_callback':RayBackend.on_success_callback,'on_failure_callback':RayBackend.on_failure_callback,...}task_args={"ray_conn_id":"ray_cluster_connection"}...@dag(default_args=default_args,..)defray_example_dag():@ray_task(**task_args)defsum_cols(df:pd.DataFrame)->pd.DataFrame:returnpd.DataFrame(df.sum()).TProject Contributors and MaintainersThis project is built in collaboration betweenAstronomerandAnyscale,
with active contributions from:Pete DeJoyDaniel ImbermanRob DeebRichard LiawCharles GreerWill DrevoThis project is formatted viablack:pipinstallblack
black.ConnectionsTBD - [Info on building a connection to Ray]