Package: prefect-ray
prefect-rayWelcome!Visit the full docshereto see additional examples and the API reference.prefect-raycontains Prefect integrations with theRayexecution framework, a flexible distributed computing framework for Python.Provides aRayTaskRunnerthat enables Prefect flows to run tasks execute tasks in parallel using Ray.Getting StartedPython setupRequires an installation of Python 3.7+.We recommend using a Python virtual environment manager such as pipenv, conda, or virtualenv.These tasks are designed to work with Prefect 2.0. For more information about how to use Prefect, please refer to thePrefect documentation.InstallationInstallprefect-raywithpip:pipinstallprefect-rayUsers running Apple Silicon (such as M1 macs) will need to install ray in a miniforge conda environment.
Clickherefor more details.Running tasks on RayTheRayTaskRunneris aPrefect task runnerthat submits tasks toRayfor parallel execution.By default, a temporary Ray instance is created for the duration of the flow run.For example, this flow counts to 3 in parallel.importtimefromprefectimportflow,taskfromprefect_rayimportRayTaskRunner@taskdefshout(number):time.sleep(0.5)print(f"#{number}")@flow(task_runner=RayTaskRunner)defcount_to(highest_number):fornumberinrange(highest_number):shout.submit(number)if__name__=="__main__":count_to(10)# outputs#3#7#2#6#4#0#1#5#8#9If you already have a Ray instance running, you can provide the connection URL via anaddressargument.To configure your flow to use theRayTaskRunner:Make sure theprefect-raycollection is installed as described earlier:pip install prefect-ray.In your flow code, importRayTaskRunnerfromprefect_ray.task_runners.Assign it as the task runner when the flow is defined using thetask_runner=RayTaskRunnerargument.For example, this flow uses theRayTaskRunnerwith a local, temporary Ray instance created by Prefect at flow run time.fromprefectimportflowfromprefect_ray.task_runnersimportRayTaskRunner@flow(task_runner=RayTaskRunner())defmy_flow():...This flow uses theRayTaskRunnerconfigured to access an existing Ray instance atray://192.0.2.255:8786.fromprefectimportflowfromprefect_ray.task_runnersimportRayTaskRunner@flow(task_runner=RayTaskRunner(address="ray://192.0.2.255:8786"))defmy_flow():...RayTaskRunneraccepts the following optional parameters:ParameterDescriptionaddressAddress of a currently running Ray instance, starting with theray://URI.init_kwargsAdditional kwargs to use when callingray.init.Note that Ray Client uses theray://URI to indicate the address of a Ray instance. If you don't provide theaddressof a Ray instance, Prefect creates a temporary instance automatically.!!! warning "Ray environment limitations"
Ray support for non-x86/64 architectures such as ARM/M1 processors with installation frompipalone and will be skipped during installation of Prefect. It is possible to manually install the blocking component withconda. See theRay documentationfor instructions.See the [Ray installation documentation](https://docs.ray.io/en/latest/ray-overview/installation.html) for further compatibility information.Running tasks on a Ray remote clusterWhen using theRayTaskRunnerwith a remote Ray cluster, you may run into issues that are not seen when using a local Ray instance. To resolve these issues, we recommend taking the following steps when working with a remote Ray cluster:By default, Prefect will not persist any data to the filesystem of the remote ray worker. However, if you want to take advantage of Prefect's caching ability, you will need to configure a remote result storage to persist results across task runs.We recommend using thePrefect UI to configure a storage blockto use for remote results storage.Here's an example of a flow that uses caching and remote result storage:fromtypingimportListfromprefectimportflow,get_run_logger,taskfromprefect.filesystemsimportS3fromprefect.tasksimporttask_input_hashfromprefect_ray.task_runnersimportRayTaskRunner# The result of this task will be cached in the configured result storage@task(cache_key_fn=task_input_hash)defsay_hello(name:str)->None:logger=get_run_logger()# This log statement will print only on the first run. Subsequent runs will be cached.logger.info(f"hello{name}!")returnname@flow(task_runner=RayTaskRunner(address="ray://<instance_public_ip_address>:10001",),# Using an S3 block that has already been created via the Prefect UIresult_storage="s3/my-result-storage",)defgreetings(names:List[str])->None:fornameinnames:say_hello.submit(name)if__name__=="__main__":greetings(["arthur","trillian","ford","marvin"])If you get an error stating that the module 'prefect' cannot be found, ensureprefectis installed on the remote cluster, with:pipinstallprefectIf you get an error with a message similar to "File system created with scheme 's3' could not be created", ensure the required Python modules are installed onboth local and remote machines. The required prerequisite modules can be found in thePrefect documentation. For example, if using S3 for the remote storage:pipinstalls3fsIf you are seeing timeout or other connection errors, double check the address provided to theRayTaskRunner. The address should look similar to:address='ray://<head_node_ip_address>:10001':RayTaskRunner(address="ray://1.23.199.255:10001")Specifying remote optionsTheremote_optionscontext can be used to control the task’s remote options.For example, we can set the number of CPUs and GPUs to use for theprocesstask:fromprefectimportflow,taskfromprefect_ray.task_runnersimportRayTaskRunnerfromprefect_ray.contextimportremote_options@taskdefprocess(x):returnx+1@flow(task_runner=RayTaskRunner())defmy_flow():# equivalent to setting @ray.remote(num_cpus=4, num_gpus=2)withremote_options(num_cpus=4,num_gpus=2):process.submit(42)ResourcesIf you encounter and bugs while usingprefect-ray, feel free to open an issue in theprefect-rayrepository.If you have any questions or issues while usingprefect-ray, you can find help in either thePrefect Discourse forumor thePrefect Slack community.Feel free to star or watchprefect-rayfor updates too!DevelopmentContributingIf you'd like to help contribute to fix an issue or add a feature toprefect-ray, pleasepropose changes through a pull request from a fork of the repository.Fork the repositoryClone the forked repositoryInstall the repository and its dependencies:pip install -e ".[dev]"Make desired changesAdd testsInsert an entry toCHANGELOG.mdInstallpre-committo perform quality checks prior to commit:pre-commit installgit commit,git push, and create a pull request