this is modal documentation an python library that can deploy cloud process your goal is to make a tools using this library we can config thelibrary we want :
Introduction to Modal
Modal lets you run code in the cloud without having to think about infrastructure.

Features
Run any code remotely within seconds.
Define container environments in code (or use one of our pre-built backends).
Scale up horizontally to thousands of containers.
Deploy and monitor persistent cron jobs.
Attach GPUs with a single line of code.
Serve your functions as web endpoints.
Use powerful primitives like distributed dictionaries and queues.
Getting started
The nicest thing about all of this is that you don’t have to set up any infrastructure. Just:

Create an account at modal.com
Install the modal Python package
Set up a token
…and you can start running jobs right away.

Modal is currently Python-only, but we may support other languages in the future.

How does it work?
Modal takes your code, puts it in a container, and executes it in the cloud.

Where does it run? Modal runs it in its own cloud environment. The benefit is that we solve all the hard infrastructure problems for you, so you don’t have to do anything. You don’t need to mess with Kubernetes, Docker or even an AWS account.
Custom containers
This guide walks you through how to define the environment your Modal functions and applications run in.

These environments are called containers. Containers are like light-weight virtual machines — container engines use operating system tricks to isolate programs from each other (“containing” them), making them work as though they were running on their own hardware with their own filesystem. This makes execution environments more reproducible, for example by preventing accidental cross-contamination of environments on the same machine. For added security, Modal runs containers using the sandboxed gVisor container runtime.

Containers are started up from a stored “snapshot” of their filesystem state called an image. Producing the image for a container is called building the image.

By default, Modal functions are executed in a Debian Linux container with a basic Python installation of the same minor version v3.x as your local Python interpreter.

Customizing this environment is critical. To make your apps and functions useful, you will probably need some third party system packages or Python libraries. To make them start up faster, you can bake data like model weights into the container image, taking advantage of Modal’s optimized filesystem for serving containers.

Modal provides a number of options to customize your container images at different levels of abstraction and granularity, from high-level convenience methods like pip_install through wrappers of core container image build features like RUN and ENV to full on “bring-your-own-Dockerfile”. We’ll cover each of these in this guide, along with tips and tricks for building images effectively when using each tool.

The typical flow for defining an image in Modal is method chaining starting from a base image, like this:

from modal import Image

image = (
    Image.debian_slim(python_version="3.10")
    .apt_install("git")
    .pip_install("torch==2.2.1")
    .env({"HALT_AND_CATCH_FIRE": 0})
    .run_commands("git clone https://github.com/modal-labs/agi && echo 'ready to go!'")
)

Copy
In addition to being Pythonic and clean, this also matches the onion-like layerwise build process of container images.

Add Python packages with pip_install
The simplest and most common container modification is to add some third party Python package, like pandas.

You can add Python packages to the environment by passing all the packages you need to the pip_install method of an image.

You can include typical Python dependency version specifiers, like "torch <= 2.0", in the arguments. But we recommend pinning dependencies tightly, like "torch == 1.9.1", to improve the reproducibility and robustness of your builds.

Of course, that means you need to start from some image. Below, we use the recommended debian_slim image as our base.

from modal import Image

datascience_image = (
    Image.debian_slim(python_version="3.10")
    .pip_install("pandas==2.2.0", "numpy")
)


@app.function(image=datascience_image)
def my_function():
    import pandas as pd
    import numpy as np

    df = pd.DataFrame()
    ...

Copy
Note that because you can define a different environment for each and every Modal function if you so choose, you don’t need to worry about virtual environment management. Containers make for much better separation of concerns!

If you want to run a specific version of Python remotely rather than just matching the one you’re running locally, provide the python_version as a string when constructing the base image, like we did above.

What if I have different Python packages locally and remotely?
You might want to use packages inside your Modal code that you don’t have on your local computer. In the example above, we build a container that uses pandas. But if we don’t have pandas locally, on the computer launching the Modal job, we can’t put import pandas at the top of the script, since it would cause an ImportError.

The easiest solution to this is to put import pandas in the function body instead, as you can see above. This means that pandas is only imported when running inside the remote Modal container, which has pandas installed.

Be careful about what you return from Modal functions that have different packages installed than the ones you have locally! Modal functions return Python objects, like pandas.DataFrames, and if your local machine doesn’t have pandas installed, it won’t be able to handle a pandas object (the error message you see will mention serialization/deserialization).

If you have a lot of functions and a lot of Python packages, you might want to keep the imports in the global scope so that every function can use the same imports. In that case, you can use the imports() context manager:

from modal import Image

pandas_image = Image.debian_slim().pip_install("pandas", "numpy")


with pandas_image.imports():
    import pandas as pd
    import numpy as np


@app.function(image=pandas_image)
def my_function():
    df = pd.DataFrame()

Copy
Note that this feature is still in beta.

Run shell commands with .run_commands
You can also supply shell commands that should be executed when building the container image.

You might use this to preload custom assets, like model parameters, so that they don’t need to be retrieved when functions start up:

from modal import Image

image_with_model = (
    Image.debian_slim().apt_install("curl").run_commands(
        "curl -O https://raw.githubusercontent.com/opencv/opencv/master/data/haarcascades/haarcascade_frontalcatface.xml",
    )
)


@app.function(image=image_with_model)
def find_cats():
    content = open("/haarcascade_frontalcatface.xml").read()
    ...

Copy
Or to make use of a hip new blazingly fast package manager:

from modal import Image

image = (
    Image.debian_slim()
    .pip_install("uv")
    .run_commands("uv pip install --system --compile torch")
)

Copy
Run a Modal function during your build with .run_function (beta)
Instead of using shell commands, you can also run a Python function as an image build step using the Image.run_function method. For example, you can use this to download model parameters from Hugging Face into your image, massively speeding up function starts:

from modal import Image, Secret

def download_models():
    import diffusers

    pipe = diffusers.StableDiffusionPipeline.from_pretrained(
        model_id, use_auth_token=os.environ["HF_TOKEN"]
    )
    pipe.save_pretrained("/model")


image = (
    Image.debian_slim()
        .pip_install("diffusers[torch]", "transformers", "ftfy", "accelerate")
        .run_function(download_models, secrets=[Secret.from_name("huggingface-secret")])
)

Copy
Any kwargs accepted by @app.function (such as Mounts, NetworkFileSystems, and specifications of resources like GPUs) can be supplied here.

Essentially, this is equivalent to running a Modal function and snapshotting the resulting filesystem as an image.

Whenever you change other features of your image, like the base image or the version of a Python package, the image will automatically be rebuilt the next time it is used. This is a bit more complicated when changing the contents of Modal functions. See the reference documentation for details.

Attach GPUs during setup
If a step in the setup of your container image should be run on an instance with a GPU (e.g., so that a package can be linked against CUDA libraries), pass a desired GPU type when defining that step:

from modal import Image

image = (
    Image.debian_slim()
    .pip_install("bitsandbytes", gpu="H100")
)

Copy
Use mamba instead of pip with micromamba_install
pip installs Python packages, but some Python workloads require the coordinated installation of system packages as well. The mamba package manager can install both. Modal provides a pre-built Micromamba base image that makes it easy to work with micromamba:

from modal import Image, App

app = App("bayes-pgm")  # Note: prior to April 2024, "app" was called "stub"

numpyro_pymc_image = (
    Image.micromamba()
    .micromamba_install("pymc==5.10.4", "numpyro==0.13.2", channels=["conda-forge"])
)


@app.function(image=numpyro_pymc_image)
def sample():
    import pymc as pm
    import numpyro as np

    print(f"Running on PyMC v{pm.__version__} with JAX/numpyro v{np.__version__} backend")
    ...

Copy
Use an existing container image with .from_registry
You don’t always need to start from scratch! Public registries like Docker Hub have many pre-built container images for common software packages.

You can use any public image in your function using Image.from_registry, so long as:

Python 3.8 or above is present, and is available as python
pip is installed correctly
The image is built for the linux/amd64 platform
The image has a valid ENTRYPOINT
from modal import Image

sklearn_image = Image.from_registry("huanjason/scikit-learn")


@app.function(image=sklearn_image)
def fit_knn():
    from sklearn.neighbors import KNeighborsClassifier
    ...

Copy
If an existing image does not have either python or pip set up properly, you can still use it. Just provide a version number as the add_python argument to install a reproducible, standalone build of Python:

from modal import Image

image1 = Image.from_registry("ubuntu:22.04", add_python="3.11")
image2 = Image.from_registry("gisops/valhalla:latest", add_python="3.11")

Copy
The from_registry method can load images from all public registries, such as Nvidia’s nvcr.io, AWS ECR, and GitHub’s ghcr.io.

We also support access to private AWS ECR and GCP Artifact Registry images.

Bring your own image definition with .from_dockerfile
Sometimes, you might be working in a setting where the environment is already defined as a container image in the form of a Dockerfile.

Modal supports defining a container image directly from a Dockerfile via the Image.from_dockerfile function. It takes a path to an existing Dockerfile.

For instance, we might write a Dockerfile based on the official Python image and adding scikit-learn:

FROM python:3.9
RUN pip install sklearn

Copy
and then define an image for Modal based on it:

from modal import Image

dockerfile_image = Image.from_dockerfile("Dockerfile")


@app.function(image=dockerfile_image)
def fit():
    import sklearn
    ...

Copy
Note that you can still do method chaining to extend this image!

Dockerfile command compatibility
Since Modal doesn’t use Docker to build containers, we have our own implementation of the Dockerfile specification. Most Dockerfiles should work out of the box, but there are some differences to be aware of.

First, a few minor Dockerfile commands and flags have not been implemented yet. Please reach out to us if your use case requires any of these.

Next, there are some command-specific things that may be useful when porting a Dockerfile to Modal.

ENTRYPOINT
While the ENTRYPOINT command is supported, there is an additional constraint to the entrypoint script provided: it must also exec the arguments passed to it at some point. This is so that Modal’s own Python entrypoint can run after your own. Most entrypoint scripts in Docker containers are wrappers over other scripts, so this is likely already the case.

If you wish to write your own entrypoint script, you can use the following as a template:

#!/usr/bin/env bash

# Your custom startup commands here.

exec "$@" # Runs the command passed to the entrypoint script.

Copy
If the above file is saved as /usr/bin/my_entrypoint.sh in your container, then you can register it as an entrypoint with ENTRYPOINT ["/usr/bin/my_entrypoint.sh"] in your Dockerfile, or with dockerfile_commands as an Image build step.

from modal import Image

image = (
    Image.debian_slim()
    .pip_install("foo")
    .dockerfile_commands('ENTRYPOINT ["/usr/bin/my_entrypoint.sh"]')
)

Copy
ENV
We currently don’t support Default value in Interpolation, such as ${VAR:-default}

Forcing an image to rebuild
Modal uses the definition of an image to determine whether it needs to be rebuilt. In some cases, you may want to force an image to rebuild, even if the definition hasn’t changed. You can do this by adding the force_build=True argument to any of the image build steps.

from modal import Image

image = (
    Image.debian_slim()
    .apt_install("git")
    .pip_install("slack-sdk", force_build=True)
    .run_commands("echo hi")
)

Copy
In the above example, both pip_install("slack-sdk") and run_commands("echo hi") will run again, but apt_install("git") will not. Remember to remove force_build=True after you’ve rebuilt the image, otherwise it will rebuild every time you run your code.
Private registries
Modal provides the Image.from_registry function, which can pull public images available from registries such as Docker Hub and GitHub Container Registry, as well as private images from registries such as AWS Elastic Container Registry (ECR), GCP Artifact Registry, and Docker Hub.

Docker Hub (Private)
To pull container images from private Docker Hub repositories, create an access token with “Read-Only” permissions and use this token value and your Docker Hub username to create a Modal Secret.

REGISTRY_USERNAME=my-dockerhub-username
REGISTRY_PASSWORD=dckr_pat_TS012345aaa67890bbbb1234ccc

Copy
Use this Secret with the modal.Image.from_registry method.

Elastic Container Registry (ECR)
You can pull images from your AWS ECR account by specifying the full image URI as follows:

from modal import Image, Secret, App

aws_secret = Secret.from_name("my-aws-secret")
image = (
    Image.from_aws_ecr(
        "000000000000.dkr.ecr.us-east-1.amazonaws.com/my-private-registry:latest",
        secret=aws_secret,
    )
    .pip_install("torch", "huggingface")
)

app = App(image=image)  # Note: prior to April 2024, "app" was called "stub"

Copy
As shown above, you also need to use a Modal Secret containing the environment variables AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_REGION. The AWS IAM user account associated with those keys must have access to the private registry you want to access.

The user needs to have the following read-only policies:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": ["ecr:GetAuthorizationToken"],
      "Effect": "Allow",
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "ecr:BatchCheckLayerAvailability",
        "ecr:GetDownloadUrlForLayer",
        "ecr:GetRepositoryPolicy",
        "ecr:DescribeRepositories",
        "ecr:ListImages",
        "ecr:DescribeImages",
        "ecr:BatchGetImage",
        "ecr:GetLifecyclePolicy",
        "ecr:GetLifecyclePolicyPreview",
        "ecr:ListTagsForResource",
        "ecr:DescribeImageScanFindings"
      ],
      "Resource": "<MY-REGISTRY-ARN>"
    }
  ]
}

Copy
You can use the IAM configuration above as a template for creating an IAM user. You can then generate an access key and create a Modal Secret using the AWS integration option. Modal will use your access keys to generate an ephemeral ECR token. That token is only used to pull image layers at the time a new image is built. We don’t store this token but will cache the image once it has been pulled.

Images on ECR must be private and follow image configuration requirements.

Google Artifact Registry and Google Container Registry
For further detail on how to pull images from Google’s image registries, see modal.Image.from_gcp_artifact_registry.
GPU acceleration
Contemporary machine learning models are large linear algebra machines, and running them with reasonable latency and throughput requires specialized hardware for executing large linear algebra tasks. The weapon of choice here is the venerable Graphics Processing Unit, or GPU.

Modal is designed from the ground up to make running your ML-powered functions on GPUs as easy, cost-effective, and performant as possible. And Modal GPUs are great for graphics too!

This guide will walk you through all the options available for running your GPU-acclerated code on Modal and suggest techniques for choosing the right hardware for your problem.

If you have code or use libraries that benefit from GPUs, you can attach the first available GPU to your function by passing the gpu="any" argument to the @app.function decorator:

import modal

app = modal.App()  # Note: prior to April 2024, "app" was called "stub"

@app.function(gpu="any")
def render_toy_story():
    # code here will be executed on a machine with an available GPU
    ...

Copy
Specifying GPU type
When gpu="any" is specified, your function runs in a container with access to a GPU. Currently this GPU will be either an NVIDIA Tesla T4 or A10G instance, and pricing is based on which one you land on.

If you need more control, you can pick a specific GPU type by changing this argument:

@app.function(gpu="A10G")
def run_sdxl_turbo():
    ...

@app.function(gpu="A100")
def run_sdxl_batch():
    ...

@app.function(gpu="H100")
def finetune_sdxl():
    ...

Copy
For information on all valid values for the gpu parameter see the reference docs.

For running, rather than training, neural networks, we recommend starting off with the A10Gs, which offer an excellent trade-off of cost and performance.

For more on how to pick a GPU for use with neural networks like LLaMA or Stable Diffusion, and for tips on how to make that GPU go brrr, check out Tim Dettemers’ blog post or the Full Stack Deep Learning page on Cloud GPUs.

Specifying GPU count
The largest machine learning models are too large to fit in the memory of just one of even the most capacious GPUs. Rather than off-loading from GPU memory to CPU memory or disk, which leads to punishing drops in latency and throughput, the usual tactic is to parallelize the model across several GPUs on the same machine — or even to distribute it across several machines, each with several GPUs.

You can run your function on a Modal machine with more than one GPU by changing the count argument in the object form of the gpu parameter:

@app.function(gpu=modal.gpu.H100(count=8))
def train_sdxl():
    ...

Copy
We also support an equivalent string-based shorthand for specifying the count:


@app.function(gpu="H100:8")
def train_sdxl():
    ...

Copy
Currently H100, A100, and T4 instances support up to 8 GPUs (up to 640 GB VRAM), and A10G instances support up to 4 GPUs (up to 96 GB VRAM). Note that requesting more than 2 GPUs per container will usually result in larger wait times. These GPUs are always attached to the same physical machine.

H100 GPUs
Modal’s fastest GPUs are the H100s, which are NVIDIA’s flagship data center chip. Modal offers H100s in the SXM form factor, with Tensor Cores capable of nearly two petaFLOPS in 16-bit precision connected via >3 TB/s bandwidth connections to 80 GB of on-device RAM. Powerful!

To request an H100, replace the gpu="any" argument with gpu="H100"

@app.function(gpu="H100")
def run_mixtral():
    ...

Copy
Check out this example to see how you can run 7B parameter language models at thousands of tokens per second using an H100 on Modal.

Before you jump for the most powerful (and so most expensive) GPU, make sure you understand where the bottlenecks are in your computations. For example, running language models with small batch sizes (e.g. one prompt at a time) results in a bottleneck on memory, not arithmetic. Since arithmetic throughput has risen faster than memory throughput in recent hardware generations, speedups for memory-bound GPU jobs are not as extreme and may not be worth the extra cost.

Additionally, it takes time for library providers to adapt to the latest hardware, so expect sharp edges, like missing CUDA kernels, when using H100s — through at least the first half of 2024.

A100 GPUs
A100s are the previous generation of top-of-the-line data center chip from NVIDIA. Modal offers two versions of the A100: one with 40 GB of RAM and another with 80 GB of RAM.

To request an A100 with 40 GB of GPU memory, replace the gpu="any" argument with gpu="A100":

@app.function(gpu="A100")
def llama_7b():
    ...

Copy
At half precision, a 34B parameter language model like LLaMA 34B will require more than 40 GB of RAM (16 bits = 2 bytes and 34 × 2 > 40). To request an 80 GB A100 that can run those models, use the object form. of the gpu argument:

@app.function(gpu=modal.gpu.A100(size="80GB"))
def llama_34b():
    ...

Copy
For the largest useful open source models, or when finetuning models that are of size 7B or higher, you may need mutliple A100s to have enough GPU RAM. Finetuning models can be particularly RAM intensive because optimizing neural networks requires storing a lot of things in memory: not only input data and weights, but also intermediate calculations, gradients, and optimizer parameters.

To use more than one GPU, set the count argument to an integer value between 2 and 8.

@app.function(gpu=modal.gpu.A100(size="80GB", count=8))
def finetune_llama_70b():
    ...
Reserving CPU and memory
Modal jobs are reserved 128 MiB of memory and 0.1 CPU cores by default. However, if there is free memory or CPU capacity on a worker, containers are free to spike above these limits.

Keep in mind that these reservations should be set to the minimum value required to have your application function correctly.

CPU cores
If you have code that must run on a larger number of cores, you can request that using the cpu argument. This allows you to specify a floating-point number of CPU cores:

import modal

app = modal.App()  # Note: prior to April 2024, "app" was called "stub"

@app.function(cpu=8.0)
def my_function():
    # code here will have access to at least 8.0 cores
    ...

Copy
Memory
If you have code that needs more guaranteed memory, you can request it using the memory argument. This expects an integer number of megabytes:

import modal

app = modal.App()  # Note: prior to April 2024, "app" was called "stub"

@app.function(memory=32768)
def my_function():
    # code here will have access to at least 32 GiB of RAM
    ...

Copy
How much can I request?
For both CPU and memory, a maximum is enforced at function creation time to ensure your application can be scheduled for execution. Requests exceeding the maximum will be rejected with an InvalidError.

As the platform grows, we plan to support larger CPU and memory reservations.

Resource limits
CPU limits
Modal containers have a soft CPU limit that is set at 4 physical cores above the CPU request. Given that the default CPU request is 0.1 cores the soft CPU limit is 4.1 cores. Above this limit the host will begin to throttle the CPU usage of the container.

Memory limits
Modal containers can have a hard memory limit which will ‘Out of Memory’ (OOM) kill containers which attempt to exceed the limit. This functionality is useful when a container has a serious memory leak. You can set the limit and have the container killed to avoid paying for the leaked GBs of memory.

mem_request = 1024
mem_limit = 2048
@app.function(
    memory=(mem_request, mem_limit),
)

Copy
Specify this limit using the memory parameter on Modal Functions.

Disk limits
Running Modal containers have access to many GBs of SSD disk, but the amount of writes is limited by:

The size of the underlying worker’s SSD disk capacity
A per-container disk quota that is set in the 100s of GBs.
Hitting either limit will cause the container’s disk writes to be rejected, which typically manifests as an OSError.
Scaling out
Modal has a few different tools that helps with increasing performance of your applications.

Parallel execution of inputs
If your code is running the same function repeatedly with different independent inputs (e.g., a grid search), the easiest way to increase performance is to run those function calls in parallel using Modal’s Function.map() method.

Here is an example if we had a function evaluate_model that takes a single argument:

import modal

app = modal.App()  # Note: prior to April 2024, "app" was called "stub"


@app.function()
def evaluate_model(x):
    ...


@app.local_entrypoint()
def main():
    inputs = list(range(100))
    for result in evaluate_model.map(inputs):  # runs many inputs in parallel
        ...

Copy
In this example, evaluate_model will be called with each of the 100 inputs (the numbers 0 - 99 in this case) roughly in parallel and the results are returned as an iterable with the results ordered in the same way as the inputs.

Exceptions
By default, if any of the function calls raises an exception, the exception will be propagated. To treat exceptions as successful results and aggregate them in the results list, pass in return_exceptions=True.

@app.function()
def my_func(a):
    if a == 2:
        raise Exception("ohno")
    return a ** 2

@app.local_entrypoint()
def main():
    print(list(my_func.map(range(3), return_exceptions=True)))
    # [0, 1, UserCodeException(Exception('ohno'))]

Copy
Starmap
If your function takes multiple variable arguments, you can either use Function.map() with one input iterator per argument, or Function.starmap() with a single input iterator containing sequences (like tuples) that can be spread over the arguments. This works similarly to Python’s built in map and itertools.starmap.

@app.function()
def my_func(a, b):
    return a + b

@app.local_entrypoint()
def main():
    assert list(my_func.starmap([(1, 2), (3, 4)])) == [3, 7]

Copy
Gotchas
Note that .map() is a method on the modal function object itself, so you don’t explicitly call the function.

Incorrect usage:

results = evaluate_model(inputs).map()

Copy
Modal’s map is also not the same as using Python’s builtin map(). While the following will technically work, it will execute all inputs in sequence rather than in parallel.

Incorrect usage:

results = map(evaluate_model, inputs)

Copy
Asynchronous usage
All Modal APIs are available in both blocking and asynchronous variants. If you are comfortable with asynchronous programming, you can use it to create arbitrary parallel execution patterns, with the added benefit that any Modal functions will be executed remotely. See the async guide or the examples for more information about asynchronous usage.

GPU acceleration
Sometimes you can speed up your applications by utilizing GPU acceleration. See the gpu section for more information.

Limiting concurrency
If you want to limit concurrency, you can use the concurrency_limit argument to app.function. For instance:

app = modal.App()  # Note: prior to April 2024, "app" was called "stub"

@app.function(concurrency_limit=5)
def f(x):
    print(x)

Copy
With this, Modal will spin up at most 5 containers at any point.
Dicts and queues
Modal provides a variety of distributed objects to enable seamless interactivity and data transfer across different components of a distributed system. Two key objects are dicts and queues, both of which serve specific roles in facilitating communication and data management in your applications.

Modal Dicts
A Dict in Modal provides distributed key-value storage. Much like a standard Python dictionary, it lets you store and retrieve values using keys. However, unlike a regular dictionary, a dict in Modal is shared across all containers of an application and can be accessed and manipulated concurrently from any of them.

import modal

app = modal.App()  # Note: prior to April 2024, "app" was called "stub"

# Create a persisted dict - the data gets retained between app runs
my_dict = modal.Dict.from_name("my-persisted-dict", create_if_missing=True)


@app.local_entrypoint()
def main():
    my_dict["key"] = "value"  # setting a value
    value = my_dict["key"]    # getting a value

Copy
Dicts in Modal are persisted, which means that the data in the dictionary is stored and can be retrieved later, even after the application is redeployed. They can also be accessed from other Modal functions.

You can store Python values of any type within Dicts, since they’re serialized using cloudpickle.

Currently, the ​overall​ size of a Dict is limited to 10 GiB. However, we intend to lower this limit as Dicts are not intended for caching large datasets. There is no per-object size limit, but the maximum number of entries per update request is 10,000.

Modal Queues
A Queue in Modal is a distributed queue-like object. It allows you to add and retrieve items in a first-in-first-out (FIFO) manner. Queues are particularly useful when you want to handle tasks or process data asynchronously, or when you need to pass messages between different components of your distributed system.

import modal

app = modal.App()  # Note: prior to April 2024, "app" was called "stub"
my_queue = modal.Queue.from_name("my-persisted-queue", create_if_missing=True)


@app.local_entrypoint()
def main():
    my_queue.put("some object")  # adding a value
    value = my_queue.get()  # retrieving a value

Copy
Similar to Dicts, Queues are also persisted and support values of any type.

Queue partitions
Queues are split into separate FIFO partitions via a string key. By default, one partition (corresponding to an empty key) is used.

A single Queue can contain up to 100,000 partitions, each with up to 5,000 items. Each item can be up to 256 KiB. These limits also apply to the default partition.

from modal import Queue, App

app = App()  # Note: prior to April 2024, "app" was called "stub"
my_queue = Queue.from_name("my-persisted-queue", create_if_missing=True)

@app.local_entrypoint()
def main():
    my_queue.put("some value")
    my_queue.put(123)

    assert my_queue.get() == "some value"
    assert my_queue.get() == 123

    my_queue.put(0)
    my_queue.put(1, partition_key="foo")
    my_queue.put(2, partition_key="bar")

    # Default and "foo" partition are ignored by the get operation.
    assert my_queue.get(partition_key="bar") == 2

    # Set custom 10s expiration time on "foo" partition.
    my_queue.put(3, partition_key="foo", partition_ttl=10)

    # (beta feature) Iterate through items in place (read immutably)
    my_queue.put(1)
    assert [v for v in my_queue.iterate()] == [0, 1]

Copy
By default, each partition is cleared 24 hours after the last put operation. A lower TTL can be specified by the partition_ttl argument in the put or put_many methods. Each partition’s expiry is handled independently.

As such, Queues are best used for communication between active functions and not relied on for persistent storage.

Asynchronous calls
Both Dicts and Queues are synchronous by default, but they support asynchronous interaction with the .aio function suffix.

@app.local_entrypoint()
async def main():
    await my_queue.put.aio(100)
    assert await my_queue.get.aio() == 100

    await my_dict.put.aio("hello", 400)
    assert await my_dict.get.aio("hello") == 400

Copy
Note that .put and .get are aliases for the overloaded indexing operators on Dicts, and you need them name for asynchronous calls.

Please see the docs on asynchronous functions for more information.

Example: Dict and Queue Interaction
To illustrate how dicts and queues can interact together in a simple distributed system, consider the following example program that crawls the web, starting from wikipedia.org and traversing links to many sites in breadth-first order. The Queue stores pages to crawl, while the Dict is used as a kill switch to stop execution of tasks immediately upon completion.

import queue
import sys
from datetime import datetime

from modal import Dict, Image, Queue, App


app = App(image=Image.debian_slim().pip_install("requests", "beautifulsoup4"))  # Note: prior to April 2024, "app" was called "stub"


def extract_links(url: str) -> list[str]:
    """Extract links from a given URL."""
    import requests
    import urllib.parse
    from bs4 import BeautifulSoup

    resp = requests.get(url, timeout=10)
    resp.raise_for_status()
    soup = BeautifulSoup(resp.text, "html.parser")
    links = []
    for link in soup.find_all("a"):
        links.append(urllib.parse.urljoin(url, link.get("href")))
    return links


@app.function()
def crawl_pages(q: Queue, d: Dict, urls: set[str]) -> None:
    for url in urls:
        if "stop" in d:
            return
        try:
            s = datetime.now()
            links = extract_links(url)
            print(f"Crawled: {url} in {datetime.now() - s}, with {len(links)} links")
            q.put_many(links)
        except Exception as exc:
            print(f"Failed to crawl: {url} with error {exc}, skipping...", file=sys.stderr)


@app.function()
def scrape(url: str):
    start_time = datetime.now()

    # Create ephemeral dicts and queues
    with Dict.ephemeral() as d, Queue.ephemeral() as q:
        # The dict is used to signal the scraping to stop
        # The queue contains the URLs that have been crawled

        # Initialize queue with a starting URL
        q.put(url)

        # Crawl until the queue is empty, or reaching some number of URLs
        visited = set()
        max_urls = 50000
        while True:
            try:
                next_urls = q.get_many(2000, timeout=5)
            except queue.Empty:
                break
            new_urls = set(next_urls) - visited
            visited |= new_urls
            if len(visited) < max_urls:
                crawl_pages.spawn(q, d, new_urls)
            else:
                d["stop"] = True

        elapsed = (datetime.now() - start_time).total_seconds()
        print(f"Crawled {len(visited)} URLs in {elapsed:.2f} seconds")


@app.local_entrypoint()
def main():
    scrape.remote("https://www.wikipedia.org/")

Copy
Starting from Wikipedia, this spawns several dozen containers (auto-scaled on demand) to crawl over 200,000 URLs in 40 seconds.

Data durability
Dict and Queue objects are backed by an in-memory database, and thus are not resilient to database server restarts. Queues and Dicts are also subject to expiration, as described by the modal.Dict and modal.Queue reference pages.

Please get in touch if you need durability for Dict or Queue objects.
Job processing
Modal can be used as a scalable job queue to handle asynchronous tasks submitted from a web app or any other Python application. This allows you to offload long-running or resource-intensive tasks to Modal, while your main application remains responsive.

Basic pattern
The basic pattern for using Modal as a job queue involves three key steps:

Defining and deploying the job processing function using modal deploy.
Submitting a job using modal.functions.Function.spawn()
Polling for the job’s result using modal.functions.FunctionCall.get()
Here’s a simple example that you can run with modal run my_job_queue.py:

# my_job_queue.py
import modal

app = modal.App("my-job-queue")

@app.function()
def process_job(data):
    # Perform the job processing here
    return {"result": data}

def submit_job(data):
    # Since the `process_job` function is deployed, need to first look it up
    process_job = modal.Function.lookup("my-job-queue", "process_job")
    call = process_job.spawn(data)
    return call.object_id

def get_job_result(call_id):
    function_call = modal.functions.FunctionCall.from_id(call_id)
    try:
        result = function_call.get(timeout=5)
    except TimeoutError:
        result = {"result": "pending"}
    return result

@app.local_entrypoint()
def main():
    data = "my-data"

    # Submit the job to Modal
    call_id = submit_job(data)
    print(get_job_result(call_id))

Copy
In this example:

process_job is the Modal function that performs the actual job processing. To deploy the process_job function on Modal, run modal deploy my_job_queue.py.
submit_job submits a new job by first looking up the deployed process_job function, then calling .spawn() with the job data. It returns the unique ID of the spawned function call.
get_job_result attempts to retrieve the result of a previously submitted job using FunctionCall.from_id() and FunctionCall.get(). FunctionCall.get() waits indefinitely by default. It takes an optional timeout argument that specifies the maximum number of seconds to wait, which can be set to 0 to poll for an output immediately. Here, if the job hasn’t completed yet, we return a pending response.
Document OCR Web App is an example that uses this pattern.

Integration with web frameworks
You can easily integrate the job queue pattern with web frameworks like FastAPI. Here’s an example, assuming that you have already deployed process_job on Modal with modal deploy as above.

import fastapi
from modal import App, asgi_app
from modal.functions import FunctionCall

app = App("fastapi-modal")
web_app = fastapi.FastAPI()

@app.function()
@asgi_app()
def fastapi_app():
    return web_app

@web_app.post("/submit")
async def submit_job_endpoint(data):
    process_job = modal.Function.lookup("my-job-queue", "process_job")

    call = process_job.spawn(data)
    return {"call_id": call.object_id}

@web_app.get("/result/{call_id}")
async def get_job_result_endpoint(call_id: str):

    function_call = FunctionCall.from_id(call_id)
    try:
        result = function_call.get(timeout=0)
    except TimeoutError:
        return fastapi.responses.JSONResponse(content="", status_code=202)

    return result

Copy
In this example:

The /submit endpoint accepts job data, submits a new job using process_job.spawn(), and returns the job’s ID to the client.
The /result/{call_id} endpoint allows the client to poll for the job’s result using the job ID. If the job hasn’t completed yet, it returns a 202 status code to indicate that the job is still being processed.
Scaling and reliability
Modal automatically scales the job queue based on the workload, spinning up new instances as needed to process jobs concurrently. It also provides built-in reliability features like automatic retries and timeout handling.

You can customize the behavior of the job queue by configuring the @app.function() decorator with options like retries, timeout, and concurrency_limit.
Managing deployments
Once you’ve finished using modal run or modal serve to iterate on your Modal code, it’s time to deploy. A Modal deployment creates and then persists an application and its objects, providing the following benefits:

Repeated application function executions will be grouped under the deployment, aiding observability and usage tracking. Programmatically triggering lots of ephemeral app runs can clutter your web and CLI interfaces.
Function calls are much faster because deployed functions are persistent and reused, not created on-demand by calls. Learn how to trigger deployed functions in Invoking deployed functions.
Scheduled functions will continue scheduling separate from any local iteration you do, and will notify you on failure.
Web endpoints keep running when you close your laptop, and their URL address matches the deployment name.
Creating deployments
Deployments are created using the modal deploy command.

 % modal deploy whisper_pod_transcriber.main
✓ Initialized. View app page at https://modal.com/apps/ap-PYc2Tb7JrkskFUI8U5w0KG.
✓ Created objects.
├── 🔨 Created populate_podcast_metadata.
├── 🔨 Mounted /home/ubuntu/whisper_pod_transcriber at /root/whisper_pod_transcriber
├── 🔨 Created fastapi_app => https://modal-labs-whisper-pod-transcriber-fastapi-app.modal.run
├── 🔨 Mounted /home/ubuntu/whisper_pod_transcriber/whisper_frontend/dist at /assets
├── 🔨 Created search_podcast.
├── 🔨 Created refresh_index.
├── 🔨 Created transcribe_segment.
├── 🔨 Created transcribe_episode..
└── 🔨 Created fetch_episodes.
✓ App deployed! 🎉

View Deployment: https://modal.com/apps/modal-labs/whisper-pod-transcriber

Copy
Running this command on an existing deployment will redeploy the app, incrementing its version. For detail on how live deployed apps transition between versions, see the Updating deployments section.

Deployments can also be created using Modal’s client library.

Viewing deployments
Deployments can be viewed either on the apps web page or by using the modal app list command.

viewing deployments in the web interface
Updating deployments
A deployment can deploy new apps or redeploy new versions of an existing deployed app. It’s useful to understand how Modal handles the transition between versions of running deployment. Modal deployments only take a few seconds, but we still ensure things run smoothly in this short deployment period.

A running deployed app will continue running and accepting requests while a deployment is happening. Existing function executions will also keep running. They will not be terminated by the deployment because they’re outdated.

However, any existing container running the old version of the app will be marked by Modal as outdated and will become ineligible to serve new requests. These outdated containers will become idle and gracefully terminate.

Any warm pool containers will also be cycled during a deployment, as the previous version’s warm pool are now outdated.

Stopping deployments
Deployed apps can be stopped in the web UI by clicking the red delete button on the deployment’s app page, or alternatively by using the modal app stop command.

Stopped deployments are eventually garbage collected.
Invoking deployed functions
Modal lets you take a function created by a deployment and call it from other contexts.

There are two ways of invoking deployed functions. If the invoking client is running Python, then the same Modal client library used to write Modal code can be used. HTTPS is used if the invoking client is not running Python and therefore cannot import the Modal client library.

Invoking with Python
Some use cases for Python invocation include:

An existing Python web server (eg. Django, Flask) wants to invoke Modal functions.
You have split your product or system into multiple Modal applications that deploy independently and call each other.
Function lookup and invocation basics
Let’s say you have a script my_shared_app.py and this script defines a Modal app with a function that computes the square of a number:

import modal

app = modal.App("my-shared-app")  # Note: prior to April 2024, "app" was called "stub"


@app.function()
def square(x: int):
    return x ** 2

Copy
You can deploy this app to create a persistent deployment:

% modal deploy shared_app.py
✓ Initialized.
✓ Created objects.
├── 🔨 Created square.
├── 🔨 Mounted /Users/erikbern/modal/shared_app.py.
✓ App deployed! 🎉

View Deployment: https://modal.com/apps/erikbern/my-shared-app

Copy
Let’s try to run this function from a different context. For instance, let’s fire up the Python interactive interpreter:

% python
Python 3.9.5 (default, May  4 2021, 03:29:30)
[Clang 12.0.0 (clang-1200.0.32.27)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import modal
>>> f = modal.Function.lookup("my-shared-app", "square")
>>> f.remote(42)
1764
>>>

Copy
This works exactly the same as a regular modal Function object. For example, you can .map() over functions invoked this way too:

>>> f = modal.Function.lookup("my-shared-app", "square")
>>> f.map([1, 2, 3, 4, 5])
[1, 4, 9, 16, 25]

Copy
Authentication
The Modal Python SDK will read the token from ~/.modal.toml which typically is created using modal token new.

Another method of providing the credentials is to set the environment variables MODAL_TOKEN_ID and MODAL_TOKEN_SECRET. If you want to call a Modal function from a context such as a web server, you can expose these environment variables to the process.

Lookup of lifecycle functions
Lifecycle functions are defined on classes, which you can look up in a different way. Consider this code:

from modal import App, enter, method

app = App("my-shared-app")  # Note: prior to April 2024, "app" was called "stub"

@app.cls()
class MyLifecycleClass:
    @enter()
    def enter(self):
        self.var = "hello world"

    @method()
    def foo(self):
        return self.var

Copy
Let’s say you deploy this app. You can then call the function by doing this:

>>> cls = modal.Cls.lookup("my-shared-app", "MyLifecycleClass")
>>> obj = cls()  # You can pass any constructor arguments here
>>> obj.foo.remote()
'hello world'

Copy
Asynchronous invocation
In certain contexts, a Modal client will need to trigger Modal functions without waiting on the result. This is done by spawning functions and receiving a FunctionCall as a handle to the triggered execution.

The following is an example of a Flask web server (running outside Modal) which accepts model training jobs to be executed within Modal. Instead of the HTTP POST request waiting on a training job to complete, which would be infeasible, the relevant Modal function is spawned and the FunctionCall object is stored for later polling of execution status.

from uuid import uuid4
from flask import Flask, jsonify, request

app = Flask(__name__)
pending_jobs = {}

...

@app.route("/jobs", methods = ["POST"])
def create_job():
    predict_fn = modal.Function.lookup("example", "train_model")
    job_id = str(uuid4())
    function_call = predict_fn.spawn(
        job_id=job_id,
        params=request.json,
    )
    pending_jobs[job_id] = function_call
    return {
        "job_id": job_id,
        "status": "pending",
    }

Copy
Importing a Modal function between Modal apps
You can also import one function defined in an app from another app:

import modal

app = modal.App("another-app")  # Note: prior to April 2024, "app" was called "stub"
app.square = modal.Function.from_name("my-shared-app", "square")


@app.function()
def cube(x):
    return x * app.square.remote(x)


@app.local_entrypoint()
def main():
    assert cube.remote(42) == 74088

Copy
Comparison with HTTPS
Compared with HTTPS invocation, Python invocation has the following benefits:

Avoids the need to create web endpoint functions.
Avoids handling serialization of request and response data between Modal and your client.
Uses the Modal client library’s built-in authentication.
Web endpoints are public to the entire internet, whereas function lookup only exposes your code to you (and your org).
You can work with shared Modal functions as if they are normal Python functions, which might be more convenient.
Invoking with HTTPS
Any non-Python application client can interact with deployed Modal applications via web endpoint functions.

Anything able to make HTTPS requests can trigger a Modal web endpoint function. Note that all deployed web endpoint functions have a stable HTTPS URL.

Some use cases for HTTPS invocation include:

Calling Modal functions from a web browser client running Javascript
Calling Modal functions from non-Python backend services (Java, Go, Ruby, NodeJS, etc)
Calling Modal functions using UNIX tools (curl, wget)
However, if the client of your Modal deployment is running Python, it’s better to use the Modal client library to invoke your Modal code.

For more detail on setting up functions for invocation over HTTP see the web endpoints guide.
Continuous deployment
It’s a common pattern to auto-deploy your Modal app as part of a CI/CD pipeline. To get you started, below is a guide to doing continuous deployment of a Modal app in GitHub.

GitHub Actions
Here’s a sample GitHub Actions workflow that deploys your app on every push to the main branch.

This requires you to create a Modal token and add it as a secret for your Github Actions workflow.

After setting up secrets, create a new workflow file in your repository at .github/workflows/ci-cd.yml with the following contents:

name: CI/CD

on:
  push:
    branches:
      - main

jobs:
  deploy:
    name: Deploy
    runs-on: ubuntu-latest
    env:
      MODAL_TOKEN_ID: ${{ secrets.MODAL_TOKEN_ID }}
      MODAL_TOKEN_SECRET: ${{ secrets.MODAL_TOKEN_SECRET }}

    steps:
      - name: Checkout Repository
        uses: actions/checkout@v4

      - name: Install Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.10"

      - name: Install Modal
        run: |
          python -m pip install --upgrade pip
          pip install modal

      - name: Deploy job
        run: |
          modal deploy my_package.my_file

Copy
Be sure to replace my_package.my_file with your actual entrypoint.

If you use multiple Modal Environments, you can additionally specify the target environment in the YAML using MODAL_ENVIRONMENT=xyz.
Scheduling remote cron jobs
A common requirement is to perform some task at a given time every day or week automatically. Modal facilitates this through function schedules.

Basic scheduling
Let’s say we have a Python module heavy.py with a function, perform_heavy_computation().

# heavy.py
def perform_heavy_computation():
    ...

@app.local_entrypoint()
def main():
    perform_heavy_computation.remote()

Copy
To schedule this function to run once per day, we create a Modal App and attach our function to it with the @app.function decorator and a schedule parameter:

# heavy.py
import modal

app = modal.App()  # Note: prior to April 2024, "app" was called "stub"

@app.function(schedule=modal.Period(days=1))
def perform_heavy_computation():
    ...

Copy
To activate the schedule, deploy your app, either through the CLI:

modal deploy --name daily_heavy heavy.py

Copy
Or programmatically:

if __name__ == "__main__":
    modal.runner.deploy_app(app)

Copy
When you make changes to your function, just rerun the deploy command to overwrite the old deployment.

Note that when you redeploy your function, modal.Period resets, and the schedule will run X hours after this most recent deployment.

If you want to run your function at a regular schedule not disturbed by deploys, modal.Cron (see below) is a better option.

Monitoring your scheduled runs
To see past execution logs for the scheduled function, go to the Apps section on the Modal web site.

Schedules currently cannot be paused. Instead the schedule should be removed and the app redeployed. Schedules can be started manually on the app’s dashboard page, using the “run now” button.

Schedule types
There are two kinds of base schedule values - modal.Period and modal.Cron.

modal.Period lets you specify an interval between function calls, e.g. Period(days=1) or Period(hours=5):


# runs once every 5 hours
@app.function(schedule=modal.Period(hours=5))
def perform_heavy_computation():
    ...

Copy
modal.Cron gives you finer control using cron syntax:

# runs at 8 am (UTC) every Monday
@app.function(schedule=modal.Cron("0 8 * * 1"))
def perform_heavy_computation():
    ...

Copy
For more details, see the API reference for Period, Cron and Function