Package: fastapi-cloud-tasks
FastAPI Cloud TasksStrongly typed background tasks with FastAPI and Google CloudTasks.Installationpip install fastapi-cloud-tasksKey featuresStrongly typed tasks.Fail at invocation site to make it easier to develop and debug.Breaking schema changes between versions will fail at task runner with Pydantic.Familiar and simple public API.delaymethod that takes same arguments as the task..schedulermethod to create recurring job.Tasks are regular FastAPI endpoints on plain old HTTP.Dependsjust works!All middlewares, telemetry, auth, debugging etc solutions for FastAPI work as is.Host task runners independent of GCP. If CloudTasks can reach the URL, it can invoke the task.Save money.Task invocation with GCP isfree for first million, then costs $0.4/million.
That's almost always cheaper than running a RabbitMQ/Redis/SQL backend for celery.Jobs cost$0.1 per job per month irrespective of invocations. 3 jobs are free.Either free or almost always cheaper than always running beat worker.If somehow, this cost ever becomes a concern, theclientcan be overriden to call any gRPC server with a compatible API.Here's a trivial emulator implementation that we will use locallyAutoscale.With a FaaS setup, your task workers can autoscale based on load.Most FaaS services have free tiers making it much cheaper than running a celery worker.How it worksDelayed jobfromfastapi_cloud_tasksimportDelayedRouteBuilderdelayed_router=APIRouter(route_class=DelayedRouteBuilder(...))classRecipe(BaseModel):ingredients:List[str]@delayed_router.post("/{restaurant}/make_dinner")asyncdefmake_dinner(restaurant:str,recipe:Recipe):# Do a ton of work here.app.include_router(delayed_router)Now we can trigger the task withmake_dinner.delay(restaurant="Taj",recipe=Recipe(ingredients=["Pav","Bhaji"]))If we want to trigger the task 30 minutes latermake_dinner.options(countdown=1800).delay(...)Scheduled Taskfromfastapi_cloud_tasksimportScheduledRouteBuilderscheduled_router=APIRouter(route_class=ScheduledRouteBuilder(...))classRecipe(BaseModel):ingredients:List[str]@scheduled_router.post("/home_cook")asyncdefhome_cook(recipe:Recipe):# Make my own foodapp.include_router(scheduled_router)# If you want to make your own breakfast every morning at 7AM IST.home_cook.scheduler(name="test-home-cook-at-7AM-IST",schedule="0 7 * * *",time_zone="Asia/Kolkata").schedule(recipe=Recipe(ingredients=["Milk","Cereal"]))ConceptCloud Tasksallows us to schedule a HTTP request in the future.FastAPImakes us define complete schema and params for an HTTP endpoint.Cloud Schedulerallows us to schedule recurring HTTP requests in the future.FastAPI Cloud Tasks works by putting the three together:GCP's Cloud Tasks + FastAPI = Partial replacement for celery's async delayed tasks.GCP's Cloud Scheduler + FastAPI = Replacement for celery beat.FastAPI Cloud Tasks + Cloud Run = Autoscaled delayed tasks.RunningLocalPre-requisites:pip install fastapi-cloud-tasksInstallcloud-tasks-emulatorAlternatively install ngrok and forward the server's portStart running the emulator in a terminalcloud-tasks-emulatorStart running the task runner on port 8000 so that it is accessible from cloud tasks.uvicornexamples.simple.main:app--reload--port8000In another terminal, trigger the task with curlcurl http://localhost:8000/triggerCheck the logs on the server, you should seeWARNING:  Hello task ran with payload: Triggered taskImportant bits of code:# complete file: examples/simple/main.py# For local, we connect to the emulator clientclient=NoneifIS_LOCAL:client=emulator_client()# Construct our DelayedRoute class with all relevant settings# This can be done once across the entire projectDelayedRoute=DelayedRouteBuilder(client=client,base_url="http://localhost:8000"queue_path=queue_path(project="gcp-project-id",location="asia-south1",queue="test-queue",),)# Override the route_class so that we can add .delay method to the endpoints and know their complete URLdelayed_router=APIRouter(route_class=DelayedRoute,prefix="/delayed")classPayload(BaseModel):message:str@delayed_router.post("/hello")asyncdefhello(p:Payload=Payload(message="Default")):logger.warning(f"Hello task ran with payload:{p.message}")# Define our app and add trigger to it.app=FastAPI()@app.get("/trigger")asyncdeftrigger():# Trigger the taskhello.delay(p=Payload(message="Triggered task"))return{"message":"Hello task triggered"}app.include_router(delayed_router)Note: You can read complete working source code of the above example inexamples/simple/main.pyIn the real world you'd have a separate process for task runner and actual task.Deployed environment / Cloud RunRunning on Cloud Run with authentication needs us to supply an OIDC token. To do that we can use ahook.Pre-requisites:Create a task queue. Copy the project id, location and queue name.Deploy the worker as a service on Cloud Run and copy it's URL.Create a service account in cloud IAM and addCloud Run Invokerrole to it.# URL of the Cloud Run servicebase_url="https://hello-randomchars-el.a.run.app"DelayedRoute=DelayedRouteBuilder(base_url=base_url,# Task queue, same as above.queue_path=queue_path(...),pre_create_hook=oidc_task_hook(token=tasks_v2.OidcToken(# Service account that you createdservice_account_email="fastapi-cloud-tasks@gcp-project-id.iam.gserviceaccount.com",audience=base_url,),),)Check the fleshed out example atexamples/full/tasks.pyIf you're not running on CloudRun and want to an OAuth Token instead, you can use theoauth_task_hookinstead.Checkfastapi_cloud_tasks/hooks.pyto get the hang od hooks and how you can use them.ConfigurationDelayedRouteBuilderUsage:DelayedRoute=DelayedRouteBuilder(...)delayed_router=APIRouter(route_class=DelayedRoute)@delayed_router.get("/simple_task")defsimple_task():return{}base_url- The URL of your worker FastAPI service.queue_path- Full path of the Cloud Tasks queue. (Hint: use the util functionqueue_path)task_create_timeout- How long should we wait before giving up on creating cloud task.pre_create_hook- If you need to edit theCreateTaskRequestbefore sending it to Cloud Tasks (eg: Auth for Cloud Run), you can do that with this hook. See hooks section below for more.client- If you need to override the Cloud Tasks client, pass the client here. (eg: changing credentials, transport etc)Task level default optionsUsage:@delayed_router.get("/simple_task")@task_default_options(...)defsimple_task():return{}All options from above can be passed askwargsto the decorator.Additional options:countdown- Seconds in the future to schedule the task.task_id- named task id for deduplication. (One task id will only be queued once.)Example:# Trigger after 5 minutes@delayed_router.get("/simple_task")@task_default_options(countdown=300)defsimple_task():return{}Delayer OptionsUsage:simple_task.options(...).delay()All options from above can be overriden per call (including DelayedRouteBuilder options likebase_url) with kwargs to theoptionsfunction before calling delay.Example:# Trigger after 2 minutessimple_task.options(countdown=120).delay()ScheduledRouteBuilderUsage:ScheduledRoute=ScheduledRouteBuilder(...)scheduled_router=APIRouter(route_class=ScheduledRoute)@scheduled_router.get("/simple_scheduled_task")defsimple_scheduled_task():return{}simple_scheduled_task.scheduler(name="simple_scheduled_task",schedule="* * * * *").schedule()HooksWe might need to override things in the task being sent to Cloud Tasks. Thepre_create_hookallows us to do that.Some hooks are included in the library.oidc_delayed_hook/oidc_scheduled_hook- Used to pass OIDC token (for Cloud Run etc).deadline_delayed_hook/deadline_scheduled_hook- Used to change the timeout for the worker of a task. (PS: this deadline is decided by the sender to the queue and not the worker)chained_hook- If you need to chain multiple hooks together, you can do that withchained_hook(hook1, hook2)Helper dependenciesmax_retries@delayed_router.post("/fail_twice",dependencies=[Depends(max_retries(2))])asyncdeffail_twice():raiseException("nooo")CloudTasksHeaders@delayed_router.get("/my_task")asyncdefmy_task(ct_headers:CloudTasksHeaders=Depends()):print(ct_headers.queue_name)Check the filefastapi_cloud_tasks/dependencies.pyfor details.ContributingRunpre-commit installon your local to get pre-commit hook.Make changes and raise a PR!If the change is massive, open an issue to discuss it before writing code.Note: This project is neither affiliated with, nor sponsored by Google.