créé moi un script qui permet de savoir si il y a des nombre premier optimisez pour modal cuda en python :
optimise le script pour modal :

Modal logo
Examples
Guide
Reference
Dashboard
Introduction
Custom container images
Custom containers
Private registries
GPUs and other resources
GPU acceleration
Reserving CPU and memory
Scaling out
Scaling out
Dicts and queues
Job processing
Concurrent inputs on a single container (beta)
Secrets and environment variables
Secrets
Environment variables
Deployment
Managing deployments
Invoke deployed functions
Continuous deployment
Scheduling and cron jobs
Web endpoints
Web endpoints
Streaming endpoints
Web endpoint URLs
Request timeouts
Tunnels (beta)
Data sharing and storage
Passing local data
Volumes
Mounting local files and directories
Storing model weights
Cloud bucket mounts (Beta)
Network file systems (superseded)
Dynamic sandboxes
Performance
Cold start performance
Memory Snapshot (beta)
Geographic latency
Reliability and robustness
Failures and retries
Preemption
Timeouts
Troubleshooting
Security and privacy
Integrations
Connecting Modal to your Vercel account
Okta SSO
Other topics
File and project structure
Developing and debugging
Modal user account setup
Workspaces
Environments
Jupyter notebooks
Asynchronous API usage
Global variables
Container lifecycle and parameters
Apps, Stubs, and entrypoints
Introduction to Modal
Modal lets you run code in the cloud without having to think about infrastructure.

Features
Run any code remotely within seconds.
Define container environments in code (or use one of our pre-built backends).
Scale up horizontally to thousands of containers.
Deploy and monitor persistent cron jobs.
Attach GPUs with a single line of code.
Serve your functions as web endpoints.
Use powerful primitives like distributed dictionaries and queues.
Getting started
The nicest thing about all of this is that you don’t have to set up any infrastructure. Just:

Create an account at modal.com
Install the modal Python package
Set up a token
…and you can start running jobs right away.

Modal is currently Python-only, but we may support other languages in the future.

How does it work?
Modal takes your code, puts it in a container, and executes it in the cloud.

Where does it run? Modal runs it in its own cloud environment. The benefit is that we solve all the hard infrastructure problems for you, so you don’t have to do anything. You don’t need to mess with Kubernetes, Docker or even an AWS account.

Introduction to Modal
Features
Getting started
How does it work?
See it in action
Hello, world! 
A simple web scraper 
Modal logo
© 2024
About
Status
Changelog
Documentation
Slack Community
Pricing
Examples
Terms
Privacy
Featured examples | Modal Docs


Modal logo
Examples
Guide
Reference
Dashboard
Introduction
Custom container images
Custom containers
Private registries
GPUs and other resources
GPU acceleration
Reserving CPU and memory
Scaling out
Scaling out
Dicts and queues
Job processing
Concurrent inputs on a single container (beta)
Secrets and environment variables
Secrets
Environment variables
Deployment
Managing deployments
Invoke deployed functions
Continuous deployment
Scheduling and cron jobs
Web endpoints
Web endpoints
Streaming endpoints
Web endpoint URLs
Request timeouts
Tunnels (beta)
Data sharing and storage
Passing local data
Volumes
Mounting local files and directories
Storing model weights
Cloud bucket mounts (Beta)
Network file systems (superseded)
Dynamic sandboxes
Performance
Cold start performance
Memory Snapshot (beta)
Geographic latency
Reliability and robustness
Failures and retries
Preemption
Timeouts
Troubleshooting
Security and privacy
Integrations
Connecting Modal to your Vercel account
Okta SSO
Other topics
File and project structure
Developing and debugging
Modal user account setup
Workspaces
Environments
Jupyter notebooks
Asynchronous API usage
Global variables
Container lifecycle and parameters
Apps, Stubs, and entrypoints
Custom containers
This guide walks you through how to define the environment your Modal functions and applications run in.

These environments are called containers. Containers are like light-weight virtual machines — container engines use operating system tricks to isolate programs from each other (“containing” them), making them work as though they were running on their own hardware with their own filesystem. This makes execution environments more reproducible, for example by preventing accidental cross-contamination of environments on the same machine. For added security, Modal runs containers using the sandboxed gVisor container runtime.

Containers are started up from a stored “snapshot” of their filesystem state called an image. Producing the image for a container is called building the image.

By default, Modal functions are executed in a Debian Linux container with a basic Python installation of the same minor version v3.x as your local Python interpreter.

Customizing this environment is critical. To make your apps and functions useful, you will probably need some third party system packages or Python libraries. To make them start up faster, you can bake data like model weights into the container image, taking advantage of Modal’s optimized filesystem for serving containers.

Modal provides a number of options to customize your container images at different levels of abstraction and granularity, from high-level convenience methods like pip_install through wrappers of core container image build features like RUN and ENV to full on “bring-your-own-Dockerfile”. We’ll cover each of these in this guide, along with tips and tricks for building images effectively when using each tool.

The typical flow for defining an image in Modal is method chaining starting from a base image, like this:

from modal import Image

image = (
    Image.debian_slim(python_version="3.10")
    .apt_install("git")
    .pip_install("torch==2.2.1")
    .env({"HALT_AND_CATCH_FIRE": 0})
    .run_commands("git clone https://github.com/modal-labs/agi && echo 'ready to go!'")
)

Copy
In addition to being Pythonic and clean, this also matches the onion-like layerwise build process of container images.

Add Python packages with pip_install
The simplest and most common container modification is to add some third party Python package, like pandas.

You can add Python packages to the environment by passing all the packages you need to the pip_install method of an image.

You can include typical Python dependency version specifiers, like "torch <= 2.0", in the arguments. But we recommend pinning dependencies tightly, like "torch == 1.9.1", to improve the reproducibility and robustness of your builds.

Of course, that means you need to start from some image. Below, we use the recommended debian_slim image as our base.

from modal import Image

datascience_image = (
    Image.debian_slim(python_version="3.10")
    .pip_install("pandas==2.2.0", "numpy")
)


@app.function(image=datascience_image)
def my_function():
    import pandas as pd
    import numpy as np

    df = pd.DataFrame()
    ...

Copy
Note that because you can define a different environment for each and every Modal function if you so choose, you don’t need to worry about virtual environment management. Containers make for much better separation of concerns!

If you want to run a specific version of Python remotely rather than just matching the one you’re running locally, provide the python_version as a string when constructing the base image, like we did above.

What if I have different Python packages locally and remotely?
You might want to use packages inside your Modal code that you don’t have on your local computer. In the example above, we build a container that uses pandas. But if we don’t have pandas locally, on the computer launching the Modal job, we can’t put import pandas at the top of the script, since it would cause an ImportError.

The easiest solution to this is to put import pandas in the function body instead, as you can see above. This means that pandas is only imported when running inside the remote Modal container, which has pandas installed.

Be careful about what you return from Modal functions that have different packages installed than the ones you have locally! Modal functions return Python objects, like pandas.DataFrames, and if your local machine doesn’t have pandas installed, it won’t be able to handle a pandas object (the error message you see will mention serialization/deserialization).

If you have a lot of functions and a lot of Python packages, you might want to keep the imports in the global scope so that every function can use the same imports. In that case, you can use the imports() context manager:

from modal import Image

pandas_image = Image.debian_slim().pip_install("pandas", "numpy")


with pandas_image.imports():
    import pandas as pd
    import numpy as np


@app.function(image=pandas_image)
def my_function():
    df = pd.DataFrame()

Copy
Note that this feature is still in beta.

Run shell commands with .run_commands
You can also supply shell commands that should be executed when building the container image.

You might use this to preload custom assets, like model parameters, so that they don’t need to be retrieved when functions start up:

from modal import Image

image_with_model = (
    Image.debian_slim().apt_install("curl").run_commands(
        "curl -O https://raw.githubusercontent.com/opencv/opencv/master/data/haarcascades/haarcascade_frontalcatface.xml",
    )
)


@app.function(image=image_with_model)
def find_cats():
    content = open("/haarcascade_frontalcatface.xml").read()
    ...

Copy
Or to make use of a hip new blazingly fast package manager:

from modal import Image

image = (
    Image.debian_slim()
    .pip_install("uv")
    .run_commands("uv pip install --system --compile torch")
)

Copy
Run a Modal function during your build with .run_function (beta)
Instead of using shell commands, you can also run a Python function as an image build step using the Image.run_function method. For example, you can use this to download model parameters from Hugging Face into your image, massively speeding up function starts:

from modal import Image, Secret

def download_models():
    import diffusers

    pipe = diffusers.StableDiffusionPipeline.from_pretrained(
        model_id, use_auth_token=os.environ["HF_TOKEN"]
    )
    pipe.save_pretrained("/model")


image = (
    Image.debian_slim()
        .pip_install("diffusers[torch]", "transformers", "ftfy", "accelerate")
        .run_function(download_models, secrets=[Secret.from_name("huggingface-secret")])
)

Copy
Any kwargs accepted by @app.function (such as Mounts, NetworkFileSystems, and specifications of resources like GPUs) can be supplied here.

Essentially, this is equivalent to running a Modal function and snapshotting the resulting filesystem as an image.

Whenever you change other features of your image, like the base image or the version of a Python package, the image will automatically be rebuilt the next time it is used. This is a bit more complicated when changing the contents of Modal functions. See the reference documentation for details.

Attach GPUs during setup
If a step in the setup of your container image should be run on an instance with a GPU (e.g., so that a package can be linked against CUDA libraries), pass a desired GPU type when defining that step:

from modal import Image

image = (
    Image.debian_slim()
    .pip_install("bitsandbytes", gpu="H100")
)

Copy
Use mamba instead of pip with micromamba_install
pip installs Python packages, but some Python workloads require the coordinated installation of system packages as well. The mamba package manager can install both. Modal provides a pre-built Micromamba base image that makes it easy to work with micromamba:

from modal import Image, App

app = App("bayes-pgm")  # Note: prior to April 2024, "app" was called "stub"

numpyro_pymc_image = (
    Image.micromamba()
    .micromamba_install("pymc==5.10.4", "numpyro==0.13.2", channels=["conda-forge"])
)


@app.function(image=numpyro_pymc_image)
def sample():
    import pymc as pm
    import numpyro as np

    print(f"Running on PyMC v{pm.__version__} with JAX/numpyro v{np.__version__} backend")
    ...

Copy
Use an existing container image with .from_registry
You don’t always need to start from scratch! Public registries like Docker Hub have many pre-built container images for common software packages.

You can use any public image in your function using Image.from_registry, so long as:

Python 3.8 or above is present, and is available as python
pip is installed correctly
The image is built for the linux/amd64 platform
The image has a valid ENTRYPOINT
from modal import Image

sklearn_image = Image.from_registry("huanjason/scikit-learn")


@app.function(image=sklearn_image)
def fit_knn():
    from sklearn.neighbors import KNeighborsClassifier
    ...

Copy
If an existing image does not have either python or pip set up properly, you can still use it. Just provide a version number as the add_python argument to install a reproducible, standalone build of Python:

from modal import Image

image1 = Image.from_registry("ubuntu:22.04", add_python="3.11")
image2 = Image.from_registry("gisops/valhalla:latest", add_python="3.11")

Copy
The from_registry method can load images from all public registries, such as Nvidia’s nvcr.io, AWS ECR, and GitHub’s ghcr.io.

We also support access to private AWS ECR and GCP Artifact Registry images.

Bring your own image definition with .from_dockerfile
Sometimes, you might be working in a setting where the environment is already defined as a container image in the form of a Dockerfile.

Modal supports defining a container image directly from a Dockerfile via the Image.from_dockerfile function. It takes a path to an existing Dockerfile.

For instance, we might write a Dockerfile based on the official Python image and adding scikit-learn:

FROM python:3.9
RUN pip install sklearn

Copy
and then define an image for Modal based on it:

from modal import Image

dockerfile_image = Image.from_dockerfile("Dockerfile")


@app.function(image=dockerfile_image)
def fit():
    import sklearn
    ...

Copy
Note that you can still do method chaining to extend this image!

Dockerfile command compatibility
Since Modal doesn’t use Docker to build containers, we have our own implementation of the Dockerfile specification. Most Dockerfiles should work out of the box, but there are some differences to be aware of.

First, a few minor Dockerfile commands and flags have not been implemented yet. Please reach out to us if your use case requires any of these.

Next, there are some command-specific things that may be useful when porting a Dockerfile to Modal.

ENTRYPOINT
While the ENTRYPOINT command is supported, there is an additional constraint to the entrypoint script provided: it must also exec the arguments passed to it at some point. This is so that Modal’s own Python entrypoint can run after your own. Most entrypoint scripts in Docker containers are wrappers over other scripts, so this is likely already the case.

If you wish to write your own entrypoint script, you can use the following as a template:

#!/usr/bin/env bash

# Your custom startup commands here.

exec "$@" # Runs the command passed to the entrypoint script.

Copy
If the above file is saved as /usr/bin/my_entrypoint.sh in your container, then you can register it as an entrypoint with ENTRYPOINT ["/usr/bin/my_entrypoint.sh"] in your Dockerfile, or with dockerfile_commands as an Image build step.

from modal import Image

image = (
    Image.debian_slim()
    .pip_install("foo")
    .dockerfile_commands('ENTRYPOINT ["/usr/bin/my_entrypoint.sh"]')
)

Copy
ENV
We currently don’t support Default value in Interpolation, such as ${VAR:-default}

Forcing an image to rebuild
Modal uses the definition of an image to determine whether it needs to be rebuilt. In some cases, you may want to force an image to rebuild, even if the definition hasn’t changed. You can do this by adding the force_build=True argument to any of the image build steps.

from modal import Image

image = (
    Image.debian_slim()
    .apt_install("git")
    .pip_install("slack-sdk", force_build=True)
    .run_commands("echo hi")
)

Copy
In the above example, both pip_install("slack-sdk") and run_commands("echo hi") will run again, but apt_install("git") will not. Remember to remove force_build=True after you’ve rebuilt the image, otherwise it will rebuild every time you run your code.

Custom containers
Add Python packages with pip_install
What if I have different Python packages locally and remotely?
Run shell commands with .run_commands
Run a Modal function during your build with .run_function (beta)
Attach GPUs during setup
Use mamba instead of pip with micromamba_install
Use an existing container image with .from_registry
Bring your own image definition with .from_dockerfile
Dockerfile command compatibility
ENTRYPOINT
ENV
Forcing an image to rebuild
See it in action
Registry image for Algolia indexing 
Modal logo
© 2024
About
Status
Changelog
Documentation
Slack Community
Pricing
Examples
Terms
Privacy
Introduction to Modal | Modal Docs

Modal logo
Examples
Guide
Reference
Dashboard
Introduction
Custom container images
Custom containers
Private registries
GPUs and other resources
GPU acceleration
Reserving CPU and memory
Scaling out
Scaling out
Dicts and queues
Job processing
Concurrent inputs on a single container (beta)
Secrets and environment variables
Secrets
Environment variables
Deployment
Managing deployments
Invoke deployed functions
Continuous deployment
Scheduling and cron jobs
Web endpoints
Web endpoints
Streaming endpoints
Web endpoint URLs
Request timeouts
Tunnels (beta)
Data sharing and storage
Passing local data
Volumes
Mounting local files and directories
Storing model weights
Cloud bucket mounts (Beta)
Network file systems (superseded)
Dynamic sandboxes
Performance
Cold start performance
Memory Snapshot (beta)
Geographic latency
Reliability and robustness
Failures and retries
Preemption
Timeouts
Troubleshooting
Security and privacy
Integrations
Connecting Modal to your Vercel account
Okta SSO
Other topics
File and project structure
Developing and debugging
Modal user account setup
Workspaces
Environments
Jupyter notebooks
Asynchronous API usage
Global variables
Container lifecycle and parameters
Apps, Stubs, and entrypoints
Custom containers
This guide walks you through how to define the environment your Modal functions and applications run in.

These environments are called containers. Containers are like light-weight virtual machines — container engines use operating system tricks to isolate programs from each other (“containing” them), making them work as though they were running on their own hardware with their own filesystem. This makes execution environments more reproducible, for example by preventing accidental cross-contamination of environments on the same machine. For added security, Modal runs containers using the sandboxed gVisor container runtime.

Containers are started up from a stored “snapshot” of their filesystem state called an image. Producing the image for a container is called building the image.

By default, Modal functions are executed in a Debian Linux container with a basic Python installation of the same minor version v3.x as your local Python interpreter.

Customizing this environment is critical. To make your apps and functions useful, you will probably need some third party system packages or Python libraries. To make them start up faster, you can bake data like model weights into the container image, taking advantage of Modal’s optimized filesystem for serving containers.

Modal provides a number of options to customize your container images at different levels of abstraction and granularity, from high-level convenience methods like pip_install through wrappers of core container image build features like RUN and ENV to full on “bring-your-own-Dockerfile”. We’ll cover each of these in this guide, along with tips and tricks for building images effectively when using each tool.

The typical flow for defining an image in Modal is method chaining starting from a base image, like this:

from modal import Image

image = (
    Image.debian_slim(python_version="3.10")
    .apt_install("git")
    .pip_install("torch==2.2.1")
    .env({"HALT_AND_CATCH_FIRE": 0})
    .run_commands("git clone https://github.com/modal-labs/agi && echo 'ready to go!'")
)

Copy
In addition to being Pythonic and clean, this also matches the onion-like layerwise build process of container images.

Add Python packages with pip_install
The simplest and most common container modification is to add some third party Python package, like pandas.

You can add Python packages to the environment by passing all the packages you need to the pip_install method of an image.

You can include typical Python dependency version specifiers, like "torch <= 2.0", in the arguments. But we recommend pinning dependencies tightly, like "torch == 1.9.1", to improve the reproducibility and robustness of your builds.

Of course, that means you need to start from some image. Below, we use the recommended debian_slim image as our base.

from modal import Image

datascience_image = (
    Image.debian_slim(python_version="3.10")
    .pip_install("pandas==2.2.0", "numpy")
)


@app.function(image=datascience_image)
def my_function():
    import pandas as pd
    import numpy as np

    df = pd.DataFrame()
    ...

Copy
Note that because you can define a different environment for each and every Modal function if you so choose, you don’t need to worry about virtual environment management. Containers make for much better separation of concerns!

If you want to run a specific version of Python remotely rather than just matching the one you’re running locally, provide the python_version as a string when constructing the base image, like we did above.

What if I have different Python packages locally and remotely?
You might want to use packages inside your Modal code that you don’t have on your local computer. In the example above, we build a container that uses pandas. But if we don’t have pandas locally, on the computer launching the Modal job, we can’t put import pandas at the top of the script, since it would cause an ImportError.

The easiest solution to this is to put import pandas in the function body instead, as you can see above. This means that pandas is only imported when running inside the remote Modal container, which has pandas installed.

Be careful about what you return from Modal functions that have different packages installed than the ones you have locally! Modal functions return Python objects, like pandas.DataFrames, and if your local machine doesn’t have pandas installed, it won’t be able to handle a pandas object (the error message you see will mention serialization/deserialization).

If you have a lot of functions and a lot of Python packages, you might want to keep the imports in the global scope so that every function can use the same imports. In that case, you can use the imports() context manager:

from modal import Image

pandas_image = Image.debian_slim().pip_install("pandas", "numpy")


with pandas_image.imports():
    import pandas as pd
    import numpy as np


@app.function(image=pandas_image)
def my_function():
    df = pd.DataFrame()

Copy
Note that this feature is still in beta.

Run shell commands with .run_commands
You can also supply shell commands that should be executed when building the container image.

You might use this to preload custom assets, like model parameters, so that they don’t need to be retrieved when functions start up:

from modal import Image

image_with_model = (
    Image.debian_slim().apt_install("curl").run_commands(
        "curl -O https://raw.githubusercontent.com/opencv/opencv/master/data/haarcascades/haarcascade_frontalcatface.xml",
    )
)


@app.function(image=image_with_model)
def find_cats():
    content = open("/haarcascade_frontalcatface.xml").read()
    ...

Copy
Or to make use of a hip new blazingly fast package manager:

from modal import Image

image = (
    Image.debian_slim()
    .pip_install("uv")
    .run_commands("uv pip install --system --compile torch")
)

Copy
Run a Modal function during your build with .run_function (beta)
Instead of using shell commands, you can also run a Python function as an image build step using the Image.run_function method. For example, you can use this to download model parameters from Hugging Face into your image, massively speeding up function starts:

from modal import Image, Secret

def download_models():
    import diffusers

    pipe = diffusers.StableDiffusionPipeline.from_pretrained(
        model_id, use_auth_token=os.environ["HF_TOKEN"]
    )
    pipe.save_pretrained("/model")


image = (
    Image.debian_slim()
        .pip_install("diffusers[torch]", "transformers", "ftfy", "accelerate")
        .run_function(download_models, secrets=[Secret.from_name("huggingface-secret")])
)

Copy
Any kwargs accepted by @app.function (such as Mounts, NetworkFileSystems, and specifications of resources like GPUs) can be supplied here.

Essentially, this is equivalent to running a Modal function and snapshotting the resulting filesystem as an image.

Whenever you change other features of your image, like the base image or the version of a Python package, the image will automatically be rebuilt the next time it is used. This is a bit more complicated when changing the contents of Modal functions. See the reference documentation for details.

Attach GPUs during setup
If a step in the setup of your container image should be run on an instance with a GPU (e.g., so that a package can be linked against CUDA libraries), pass a desired GPU type when defining that step:

from modal import Image

image = (
    Image.debian_slim()
    .pip_install("bitsandbytes", gpu="H100")
)

Copy
Use mamba instead of pip with micromamba_install
pip installs Python packages, but some Python workloads require the coordinated installation of system packages as well. The mamba package manager can install both. Modal provides a pre-built Micromamba base image that makes it easy to work with micromamba:

from modal import Image, App

app = App("bayes-pgm")  # Note: prior to April 2024, "app" was called "stub"

numpyro_pymc_image = (
    Image.micromamba()
    .micromamba_install("pymc==5.10.4", "numpyro==0.13.2", channels=["conda-forge"])
)


@app.function(image=numpyro_pymc_image)
def sample():
    import pymc as pm
    import numpyro as np

    print(f"Running on PyMC v{pm.__version__} with JAX/numpyro v{np.__version__} backend")
    ...

Copy
Use an existing container image with .from_registry
You don’t always need to start from scratch! Public registries like Docker Hub have many pre-built container images for common software packages.

You can use any public image in your function using Image.from_registry, so long as:

Python 3.8 or above is present, and is available as python
pip is installed correctly
The image is built for the linux/amd64 platform
The image has a valid ENTRYPOINT
from modal import Image

sklearn_image = Image.from_registry("huanjason/scikit-learn")


@app.function(image=sklearn_image)
def fit_knn():
    from sklearn.neighbors import KNeighborsClassifier
    ...

Copy
If an existing image does not have either python or pip set up properly, you can still use it. Just provide a version number as the add_python argument to install a reproducible, standalone build of Python:

from modal import Image

image1 = Image.from_registry("ubuntu:22.04", add_python="3.11")
image2 = Image.from_registry("gisops/valhalla:latest", add_python="3.11")

Copy
The from_registry method can load images from all public registries, such as Nvidia’s nvcr.io, AWS ECR, and GitHub’s ghcr.io.

We also support access to private AWS ECR and GCP Artifact Registry images.

Bring your own image definition with .from_dockerfile
Sometimes, you might be working in a setting where the environment is already defined as a container image in the form of a Dockerfile.

Modal supports defining a container image directly from a Dockerfile via the Image.from_dockerfile function. It takes a path to an existing Dockerfile.

For instance, we might write a Dockerfile based on the official Python image and adding scikit-learn:

FROM python:3.9
RUN pip install sklearn

Copy
and then define an image for Modal based on it:

from modal import Image

dockerfile_image = Image.from_dockerfile("Dockerfile")


@app.function(image=dockerfile_image)
def fit():
    import sklearn
    ...

Copy
Note that you can still do method chaining to extend this image!

Dockerfile command compatibility
Since Modal doesn’t use Docker to build containers, we have our own implementation of the Dockerfile specification. Most Dockerfiles should work out of the box, but there are some differences to be aware of.

First, a few minor Dockerfile commands and flags have not been implemented yet. Please reach out to us if your use case requires any of these.

Next, there are some command-specific things that may be useful when porting a Dockerfile to Modal.

ENTRYPOINT
While the ENTRYPOINT command is supported, there is an additional constraint to the entrypoint script provided: it must also exec the arguments passed to it at some point. This is so that Modal’s own Python entrypoint can run after your own. Most entrypoint scripts in Docker containers are wrappers over other scripts, so this is likely already the case.

If you wish to write your own entrypoint script, you can use the following as a template:

#!/usr/bin/env bash

# Your custom startup commands here.

exec "$@" # Runs the command passed to the entrypoint script.

Copy
If the above file is saved as /usr/bin/my_entrypoint.sh in your container, then you can register it as an entrypoint with ENTRYPOINT ["/usr/bin/my_entrypoint.sh"] in your Dockerfile, or with dockerfile_commands as an Image build step.

from modal import Image

image = (
    Image.debian_slim()
    .pip_install("foo")
    .dockerfile_commands('ENTRYPOINT ["/usr/bin/my_entrypoint.sh"]')
)

Copy
ENV
We currently don’t support Default value in Interpolation, such as ${VAR:-default}

Forcing an image to rebuild
Modal uses the definition of an image to determine whether it needs to be rebuilt. In some cases, you may want to force an image to rebuild, even if the definition hasn’t changed. You can do this by adding the force_build=True argument to any of the image build steps.

from modal import Image

image = (
    Image.debian_slim()
    .apt_install("git")
    .pip_install("slack-sdk", force_build=True)
    .run_commands("echo hi")
)

Copy
In the above example, both pip_install("slack-sdk") and run_commands("echo hi") will run again, but apt_install("git") will not. Remember to remove force_build=True after you’ve rebuilt the image, otherwise it will rebuild every time you run your code.

Custom containers
Add Python packages with pip_install
What if I have different Python packages locally and remotely?
Run shell commands with .run_commands
Run a Modal function during your build with .run_function (beta)
Attach GPUs during setup
Use mamba instead of pip with micromamba_install
Use an existing container image with .from_registry
Bring your own image definition with .from_dockerfile
Dockerfile command compatibility
ENTRYPOINT
ENV
Forcing an image to rebuild
See it in action
Registry image for Algolia indexing 
Modal logo
© 2024
About
Status
Changelog
Documentation
Slack Community
Pricing
Examples
Terms
Privacy
Introduction to Modal | Modal Docs

Modal logo
Examples
Guide
Reference
Dashboard
Introduction
Custom container images
Custom containers
Private registries
GPUs and other resources
GPU acceleration
Reserving CPU and memory
Scaling out
Scaling out
Dicts and queues
Job processing
Concurrent inputs on a single container (beta)
Secrets and environment variables
Secrets
Environment variables
Deployment
Managing deployments
Invoke deployed functions
Continuous deployment
Scheduling and cron jobs
Web endpoints
Web endpoints
Streaming endpoints
Web endpoint URLs
Request timeouts
Tunnels (beta)
Data sharing and storage
Passing local data
Volumes
Mounting local files and directories
Storing model weights
Cloud bucket mounts (Beta)
Network file systems (superseded)
Dynamic sandboxes
Performance
Cold start performance
Memory Snapshot (beta)
Geographic latency
Reliability and robustness
Failures and retries
Preemption
Timeouts
Troubleshooting
Security and privacy
Integrations
Connecting Modal to your Vercel account
Okta SSO
Other topics
File and project structure
Developing and debugging
Modal user account setup
Workspaces
Environments
Jupyter notebooks
Asynchronous API usage
Global variables
Container lifecycle and parameters
Apps, Stubs, and entrypoints
Custom containers
This guide walks you through how to define the environment your Modal functions and applications run in.

These environments are called containers. Containers are like light-weight virtual machines — container engines use operating system tricks to isolate programs from each other (“containing” them), making them work as though they were running on their own hardware with their own filesystem. This makes execution environments more reproducible, for example by preventing accidental cross-contamination of environments on the same machine. For added security, Modal runs containers using the sandboxed gVisor container runtime.

Containers are started up from a stored “snapshot” of their filesystem state called an image. Producing the image for a container is called building the image.

By default, Modal functions are executed in a Debian Linux container with a basic Python installation of the same minor version v3.x as your local Python interpreter.

Customizing this environment is critical. To make your apps and functions useful, you will probably need some third party system packages or Python libraries. To make them start up faster, you can bake data like model weights into the container image, taking advantage of Modal’s optimized filesystem for serving containers.

Modal provides a number of options to customize your container images at different levels of abstraction and granularity, from high-level convenience methods like pip_install through wrappers of core container image build features like RUN and ENV to full on “bring-your-own-Dockerfile”. We’ll cover each of these in this guide, along with tips and tricks for building images effectively when using each tool.

The typical flow for defining an image in Modal is method chaining starting from a base image, like this:

from modal import Image

image = (
    Image.debian_slim(python_version="3.10")
    .apt_install("git")
    .pip_install("torch==2.2.1")
    .env({"HALT_AND_CATCH_FIRE": 0})
    .run_commands("git clone https://github.com/modal-labs/agi && echo 'ready to go!'")
)

Copy
In addition to being Pythonic and clean, this also matches the onion-like layerwise build process of container images.

Add Python packages with pip_install
The simplest and most common container modification is to add some third party Python package, like pandas.

You can add Python packages to the environment by passing all the packages you need to the pip_install method of an image.

You can include typical Python dependency version specifiers, like "torch <= 2.0", in the arguments. But we recommend pinning dependencies tightly, like "torch == 1.9.1", to improve the reproducibility and robustness of your builds.

Of course, that means you need to start from some image. Below, we use the recommended debian_slim image as our base.

from modal import Image

datascience_image = (
    Image.debian_slim(python_version="3.10")
    .pip_install("pandas==2.2.0", "numpy")
)


@app.function(image=datascience_image)
def my_function():
    import pandas as pd
    import numpy as np

    df = pd.DataFrame()
    ...

Copy
Note that because you can define a different environment for each and every Modal function if you so choose, you don’t need to worry about virtual environment management. Containers make for much better separation of concerns!

If you want to run a specific version of Python remotely rather than just matching the one you’re running locally, provide the python_version as a string when constructing the base image, like we did above.

What if I have different Python packages locally and remotely?
You might want to use packages inside your Modal code that you don’t have on your local computer. In the example above, we build a container that uses pandas. But if we don’t have pandas locally, on the computer launching the Modal job, we can’t put import pandas at the top of the script, since it would cause an ImportError.

The easiest solution to this is to put import pandas in the function body instead, as you can see above. This means that pandas is only imported when running inside the remote Modal container, which has pandas installed.

Be careful about what you return from Modal functions that have different packages installed than the ones you have locally! Modal functions return Python objects, like pandas.DataFrames, and if your local machine doesn’t have pandas installed, it won’t be able to handle a pandas object (the error message you see will mention serialization/deserialization).

If you have a lot of functions and a lot of Python packages, you might want to keep the imports in the global scope so that every function can use the same imports. In that case, you can use the imports() context manager:

from modal import Image

pandas_image = Image.debian_slim().pip_install("pandas", "numpy")


with pandas_image.imports():
    import pandas as pd
    import numpy as np


@app.function(image=pandas_image)
def my_function():
    df = pd.DataFrame()

Copy
Note that this feature is still in beta.

Run shell commands with .run_commands
You can also supply shell commands that should be executed when building the container image.

You might use this to preload custom assets, like model parameters, so that they don’t need to be retrieved when functions start up:

from modal import Image

image_with_model = (
    Image.debian_slim().apt_install("curl").run_commands(
        "curl -O https://raw.githubusercontent.com/opencv/opencv/master/data/haarcascades/haarcascade_frontalcatface.xml",
    )
)


@app.function(image=image_with_model)
def find_cats():
    content = open("/haarcascade_frontalcatface.xml").read()
    ...

Copy
Or to make use of a hip new blazingly fast package manager:

from modal import Image

image = (
    Image.debian_slim()
    .pip_install("uv")
    .run_commands("uv pip install --system --compile torch")
)

Copy
Run a Modal function during your build with .run_function (beta)
Instead of using shell commands, you can also run a Python function as an image build step using the Image.run_function method. For example, you can use this to download model parameters from Hugging Face into your image, massively speeding up function starts:

from modal import Image, Secret

def download_models():
    import diffusers

    pipe = diffusers.StableDiffusionPipeline.from_pretrained(
        model_id, use_auth_token=os.environ["HF_TOKEN"]
    )
    pipe.save_pretrained("/model")


image = (
    Image.debian_slim()
        .pip_install("diffusers[torch]", "transformers", "ftfy", "accelerate")
        .run_function(download_models, secrets=[Secret.from_name("huggingface-secret")])
)

Copy
Any kwargs accepted by @app.function (such as Mounts, NetworkFileSystems, and specifications of resources like GPUs) can be supplied here.

Essentially, this is equivalent to running a Modal function and snapshotting the resulting filesystem as an image.

Whenever you change other features of your image, like the base image or the version of a Python package, the image will automatically be rebuilt the next time it is used. This is a bit more complicated when changing the contents of Modal functions. See the reference documentation for details.

Attach GPUs during setup
If a step in the setup of your container image should be run on an instance with a GPU (e.g., so that a package can be linked against CUDA libraries), pass a desired GPU type when defining that step:

from modal import Image

image = (
    Image.debian_slim()
    .pip_install("bitsandbytes", gpu="H100")
)

Copy
Use mamba instead of pip with micromamba_install
pip installs Python packages, but some Python workloads require the coordinated installation of system packages as well. The mamba package manager can install both. Modal provides a pre-built Micromamba base image that makes it easy to work with micromamba:

from modal import Image, App

app = App("bayes-pgm")  # Note: prior to April 2024, "app" was called "stub"

numpyro_pymc_image = (
    Image.micromamba()
    .micromamba_install("pymc==5.10.4", "numpyro==0.13.2", channels=["conda-forge"])
)


@app.function(image=numpyro_pymc_image)
def sample():
    import pymc as pm
    import numpyro as np

    print(f"Running on PyMC v{pm.__version__} with JAX/numpyro v{np.__version__} backend")
    ...

Copy
Use an existing container image with .from_registry
You don’t always need to start from scratch! Public registries like Docker Hub have many pre-built container images for common software packages.

You can use any public image in your function using Image.from_registry, so long as:

Python 3.8 or above is present, and is available as python
pip is installed correctly
The image is built for the linux/amd64 platform
The image has a valid ENTRYPOINT
from modal import Image

sklearn_image = Image.from_registry("huanjason/scikit-learn")


@app.function(image=sklearn_image)
def fit_knn():
    from sklearn.neighbors import KNeighborsClassifier
    ...

Copy
If an existing image does not have either python or pip set up properly, you can still use it. Just provide a version number as the add_python argument to install a reproducible, standalone build of Python:

from modal import Image

image1 = Image.from_registry("ubuntu:22.04", add_python="3.11")
image2 = Image.from_registry("gisops/valhalla:latest", add_python="3.11")

Copy
The from_registry method can load images from all public registries, such as Nvidia’s nvcr.io, AWS ECR, and GitHub’s ghcr.io.

We also support access to private AWS ECR and GCP Artifact Registry images.

Bring your own image definition with .from_dockerfile
Sometimes, you might be working in a setting where the environment is already defined as a container image in the form of a Dockerfile.

Modal supports defining a container image directly from a Dockerfile via the Image.from_dockerfile function. It takes a path to an existing Dockerfile.

For instance, we might write a Dockerfile based on the official Python image and adding scikit-learn:

FROM python:3.9
RUN pip install sklearn

Copy
and then define an image for Modal based on it:

from modal import Image

dockerfile_image = Image.from_dockerfile("Dockerfile")


@app.function(image=dockerfile_image)
def fit():
    import sklearn
    ...

Copy
Note that you can still do method chaining to extend this image!

Dockerfile command compatibility
Since Modal doesn’t use Docker to build containers, we have our own implementation of the Dockerfile specification. Most Dockerfiles should work out of the box, but there are some differences to be aware of.

First, a few minor Dockerfile commands and flags have not been implemented yet. Please reach out to us if your use case requires any of these.

Next, there are some command-specific things that may be useful when porting a Dockerfile to Modal.

ENTRYPOINT
While the ENTRYPOINT command is supported, there is an additional constraint to the entrypoint script provided: it must also exec the arguments passed to it at some point. This is so that Modal’s own Python entrypoint can run after your own. Most entrypoint scripts in Docker containers are wrappers over other scripts, so this is likely already the case.

If you wish to write your own entrypoint script, you can use the following as a template:

#!/usr/bin/env bash

# Your custom startup commands here.

exec "$@" # Runs the command passed to the entrypoint script.

Copy
If the above file is saved as /usr/bin/my_entrypoint.sh in your container, then you can register it as an entrypoint with ENTRYPOINT ["/usr/bin/my_entrypoint.sh"] in your Dockerfile, or with dockerfile_commands as an Image build step.

from modal import Image

image = (
    Image.debian_slim()
    .pip_install("foo")
    .dockerfile_commands('ENTRYPOINT ["/usr/bin/my_entrypoint.sh"]')
)

Copy
ENV
We currently don’t support Default value in Interpolation, such as ${VAR:-default}

Forcing an image to rebuild
Modal uses the definition of an image to determine whether it needs to be rebuilt. In some cases, you may want to force an image to rebuild, even if the definition hasn’t changed. You can do this by adding the force_build=True argument to any of the image build steps.

from modal import Image

image = (
    Image.debian_slim()
    .apt_install("git")
    .pip_install("slack-sdk", force_build=True)
    .run_commands("echo hi")
)

Copy
In the above example, both pip_install("slack-sdk") and run_commands("echo hi") will run again, but apt_install("git") will not. Remember to remove force_build=True after you’ve rebuilt the image, otherwise it will rebuild every time you run your code.

Custom containers
Add Python packages with pip_install
What if I have different Python packages locally and remotely?
Run shell commands with .run_commands
Run a Modal function during your build with .run_function (beta)
Attach GPUs during setup
Use mamba instead of pip with micromamba_install
Use an existing container image with .from_registry
Bring your own image definition with .from_dockerfile
Dockerfile command compatibility
ENTRYPOINT
ENV
Forcing an image to rebuild
See it in action
Registry image for Algolia indexing 
Modal logo
© 2024
About
Status
Changelog
Documentation
Slack Community
Pricing
Examples
Terms
Privacy
Introduction to Modal | Modal Docs

Modal logo
Examples
Guide
Reference
Dashboard
Introduction
Custom container images
Custom containers
Private registries
GPUs and other resources
GPU acceleration
Reserving CPU and memory
Scaling out
Scaling out
Dicts and queues
Job processing
Concurrent inputs on a single container (beta)
Secrets and environment variables
Secrets
Environment variables
Deployment
Managing deployments
Invoke deployed functions
Continuous deployment
Scheduling and cron jobs
Web endpoints
Web endpoints
Streaming endpoints
Web endpoint URLs
Request timeouts
Tunnels (beta)
Data sharing and storage
Passing local data
Volumes
Mounting local files and directories
Storing model weights
Cloud bucket mounts (Beta)
Network file systems (superseded)
Dynamic sandboxes
Performance
Cold start performance
Memory Snapshot (beta)
Geographic latency
Reliability and robustness
Failures and retries
Preemption
Timeouts
Troubleshooting
Security and privacy
Integrations
Connecting Modal to your Vercel account
Okta SSO
Other topics
File and project structure
Developing and debugging
Modal user account setup
Workspaces
Environments
Jupyter notebooks
Asynchronous API usage
Global variables
Container lifecycle and parameters
Apps, Stubs, and entrypoints
Reserving CPU and memory
Modal jobs are reserved 128 MiB of memory and 0.1 CPU cores by default. However, if there is free memory or CPU capacity on a worker, containers are free to spike above these limits.

Keep in mind that these reservations should be set to the minimum value required to have your application function correctly.

CPU cores
If you have code that must run on a larger number of cores, you can request that using the cpu argument. This allows you to specify a floating-point number of CPU cores:

import modal

app = modal.App()  # Note: prior to April 2024, "app" was called "stub"

@app.function(cpu=8.0)
def my_function():
    # code here will have access to at least 8.0 cores
    ...

Copy
Memory
If you have code that needs more guaranteed memory, you can request it using the memory argument. This expects an integer number of megabytes:

import modal

app = modal.App()  # Note: prior to April 2024, "app" was called "stub"

@app.function(memory=32768)
def my_function():
    # code here will have access to at least 32 GiB of RAM
    ...

Copy
How much can I request?
For both CPU and memory, a maximum is enforced at function creation time to ensure your application can be scheduled for execution. Requests exceeding the maximum will be rejected with an InvalidError.

As the platform grows, we plan to support larger CPU and memory reservations.

Resource limits
CPU limits
Modal containers have a soft CPU limit that is set at 4 physical cores above the CPU request. Given that the default CPU request is 0.1 cores the soft CPU limit is 4.1 cores. Above this limit the host will begin to throttle the CPU usage of the container.

Memory limits
Modal containers can have a hard memory limit which will ‘Out of Memory’ (OOM) kill containers which attempt to exceed the limit. This functionality is useful when a container has a serious memory leak. You can set the limit and have the container killed to avoid paying for the leaked GBs of memory.

mem_request = 1024
mem_limit = 2048
@app.function(
    memory=(mem_request, mem_limit),
)

Copy
Specify this limit using the memory parameter on Modal Functions.

Disk limits
Running Modal containers have access to many GBs of SSD disk, but the amount of writes is limited by:

The size of the underlying worker’s SSD disk capacity
A per-container disk quota that is set in the 100s of GBs.
Hitting either limit will cause the container’s disk writes to be rejected, which typically manifests as an OSError.

Reserving CPU and memory
CPU cores
Memory
How much can I request?
Resource limits
CPU limits
Memory limits
Disk limits
Modal logo
© 2024
About
Status
Changelog
Documentation
Slack Community
Pricing
Examples
Terms
Privacy
GPU acceleration | Modal Docs

Modal logo
Examples
Guide
Reference
Dashboard
Introduction
Custom container images
Custom containers
Private registries
GPUs and other resources
GPU acceleration
Reserving CPU and memory
Scaling out
Scaling out
Dicts and queues
Job processing
Concurrent inputs on a single container (beta)
Secrets and environment variables
Secrets
Environment variables
Deployment
Managing deployments
Invoke deployed functions
Continuous deployment
Scheduling and cron jobs
Web endpoints
Web endpoints
Streaming endpoints
Web endpoint URLs
Request timeouts
Tunnels (beta)
Data sharing and storage
Passing local data
Volumes
Mounting local files and directories
Storing model weights
Cloud bucket mounts (Beta)
Network file systems (superseded)
Dynamic sandboxes
Performance
Cold start performance
Memory Snapshot (beta)
Geographic latency
Reliability and robustness
Failures and retries
Preemption
Timeouts
Troubleshooting
Security and privacy
Integrations
Connecting Modal to your Vercel account
Okta SSO
Other topics
File and project structure
Developing and debugging
Modal user account setup
Workspaces
Environments
Jupyter notebooks
Asynchronous API usage
Global variables
Container lifecycle and parameters
Apps, Stubs, and entrypoints
Scaling out
Modal has a few different tools that helps with increasing performance of your applications.

Parallel execution of inputs
If your code is running the same function repeatedly with different independent inputs (e.g., a grid search), the easiest way to increase performance is to run those function calls in parallel using Modal’s Function.map() method.

Here is an example if we had a function evaluate_model that takes a single argument:

import modal

app = modal.App()  # Note: prior to April 2024, "app" was called "stub"


@app.function()
def evaluate_model(x):
    ...


@app.local_entrypoint()
def main():
    inputs = list(range(100))
    for result in evaluate_model.map(inputs):  # runs many inputs in parallel
        ...

Copy
In this example, evaluate_model will be called with each of the 100 inputs (the numbers 0 - 99 in this case) roughly in parallel and the results are returned as an iterable with the results ordered in the same way as the inputs.

Exceptions
By default, if any of the function calls raises an exception, the exception will be propagated. To treat exceptions as successful results and aggregate them in the results list, pass in return_exceptions=True.

@app.function()
def my_func(a):
    if a == 2:
        raise Exception("ohno")
    return a ** 2

@app.local_entrypoint()
def main():
    print(list(my_func.map(range(3), return_exceptions=True)))
    # [0, 1, UserCodeException(Exception('ohno'))]

Copy
Starmap
If your function takes multiple variable arguments, you can either use Function.map() with one input iterator per argument, or Function.starmap() with a single input iterator containing sequences (like tuples) that can be spread over the arguments. This works similarly to Python’s built in map and itertools.starmap.

@app.function()
def my_func(a, b):
    return a + b

@app.local_entrypoint()
def main():
    assert list(my_func.starmap([(1, 2), (3, 4)])) == [3, 7]

Copy
Gotchas
Note that .map() is a method on the modal function object itself, so you don’t explicitly call the function.

Incorrect usage:

results = evaluate_model(inputs).map()

Copy
Modal’s map is also not the same as using Python’s builtin map(). While the following will technically work, it will execute all inputs in sequence rather than in parallel.

Incorrect usage:

results = map(evaluate_model, inputs)

Copy
Asynchronous usage
All Modal APIs are available in both blocking and asynchronous variants. If you are comfortable with asynchronous programming, you can use it to create arbitrary parallel execution patterns, with the added benefit that any Modal functions will be executed remotely. See the async guide or the examples for more information about asynchronous usage.

GPU acceleration
Sometimes you can speed up your applications by utilizing GPU acceleration. See the gpu section for more information.

Limiting concurrency
If you want to limit concurrency, you can use the concurrency_limit argument to app.function. For instance:

app = modal.App()  # Note: prior to April 2024, "app" was called "stub"

@app.function(concurrency_limit=5)
def f(x):
    print(x)

Copy
With this, Modal will spin up at most 5 containers at any point.

Scaling out
Parallel execution of inputs
Exceptions
Starmap
Gotchas
Asynchronous usage
GPU acceleration
Limiting concurrency
See it in action
Auto-scaling LLM inference endpoints 
Job queue for OCR 
Parallel web scraping 
Modal logo
© 2024
About
Status
Changelog
Documentation
Slack Community
Pricing
Examples
Terms
Privacy
Reserving CPU and memory | Modal Docs

Modal logo
Examples
Guide
Reference
Dashboard
Introduction
Custom container images
Custom containers
Private registries
GPUs and other resources
GPU acceleration
Reserving CPU and memory
Scaling out
Scaling out
Dicts and queues
Job processing
Concurrent inputs on a single container (beta)
Secrets and environment variables
Secrets
Environment variables
Deployment
Managing deployments
Invoke deployed functions
Continuous deployment
Scheduling and cron jobs
Web endpoints
Web endpoints
Streaming endpoints
Web endpoint URLs
Request timeouts
Tunnels (beta)
Data sharing and storage
Passing local data
Volumes
Mounting local files and directories
Storing model weights
Cloud bucket mounts (Beta)
Network file systems (superseded)
Dynamic sandboxes
Performance
Cold start performance
Memory Snapshot (beta)
Geographic latency
Reliability and robustness
Failures and retries
Preemption
Timeouts
Troubleshooting
Security and privacy
Integrations
Connecting Modal to your Vercel account
Okta SSO
Other topics
File and project structure
Developing and debugging
Modal user account setup
Workspaces
Environments
Jupyter notebooks
Asynchronous API usage
Global variables
Container lifecycle and parameters
Apps, Stubs, and entrypoints
Dicts and queues
Modal provides a variety of distributed objects to enable seamless interactivity and data transfer across different components of a distributed system. Two key objects are dicts and queues, both of which serve specific roles in facilitating communication and data management in your applications.

Modal Dicts
A Dict in Modal provides distributed key-value storage. Much like a standard Python dictionary, it lets you store and retrieve values using keys. However, unlike a regular dictionary, a dict in Modal is shared across all containers of an application and can be accessed and manipulated concurrently from any of them.

import modal

app = modal.App()  # Note: prior to April 2024, "app" was called "stub"

# Create a persisted dict - the data gets retained between app runs
my_dict = modal.Dict.from_name("my-persisted-dict", create_if_missing=True)


@app.local_entrypoint()
def main():
    my_dict["key"] = "value"  # setting a value
    value = my_dict["key"]    # getting a value

Copy
Dicts in Modal are persisted, which means that the data in the dictionary is stored and can be retrieved later, even after the application is redeployed. They can also be accessed from other Modal functions.

You can store Python values of any type within Dicts, since they’re serialized using cloudpickle.

Currently, the ​overall​ size of a Dict is limited to 10 GiB. However, we intend to lower this limit as Dicts are not intended for caching large datasets. There is no per-object size limit, but the maximum number of entries per update request is 10,000.

Modal Queues
A Queue in Modal is a distributed queue-like object. It allows you to add and retrieve items in a first-in-first-out (FIFO) manner. Queues are particularly useful when you want to handle tasks or process data asynchronously, or when you need to pass messages between different components of your distributed system.

import modal

app = modal.App()  # Note: prior to April 2024, "app" was called "stub"
my_queue = modal.Queue.from_name("my-persisted-queue", create_if_missing=True)


@app.local_entrypoint()
def main():
    my_queue.put("some object")  # adding a value
    value = my_queue.get()  # retrieving a value

Copy
Similar to Dicts, Queues are also persisted and support values of any type.

Queue partitions
Queues are split into separate FIFO partitions via a string key. By default, one partition (corresponding to an empty key) is used.

A single Queue can contain up to 100,000 partitions, each with up to 5,000 items. Each item can be up to 256 KiB. These limits also apply to the default partition.

from modal import Queue, App

app = App()  # Note: prior to April 2024, "app" was called "stub"
my_queue = Queue.from_name("my-persisted-queue", create_if_missing=True)

@app.local_entrypoint()
def main():
    my_queue.put("some value")
    my_queue.put(123)

    assert my_queue.get() == "some value"
    assert my_queue.get() == 123

    my_queue.put(0)
    my_queue.put(1, partition_key="foo")
    my_queue.put(2, partition_key="bar")

    # Default and "foo" partition are ignored by the get operation.
    assert my_queue.get(partition_key="bar") == 2

    # Set custom 10s expiration time on "foo" partition.
    my_queue.put(3, partition_key="foo", partition_ttl=10)

    # (beta feature) Iterate through items in place (read immutably)
    my_queue.put(1)
    assert [v for v in my_queue.iterate()] == [0, 1]

Copy
By default, each partition is cleared 24 hours after the last put operation. A lower TTL can be specified by the partition_ttl argument in the put or put_many methods. Each partition’s expiry is handled independently.

As such, Queues are best used for communication between active functions and not relied on for persistent storage.

Asynchronous calls
Both Dicts and Queues are synchronous by default, but they support asynchronous interaction with the .aio function suffix.

@app.local_entrypoint()
async def main():
    await my_queue.put.aio(100)
    assert await my_queue.get.aio() == 100

    await my_dict.put.aio("hello", 400)
    assert await my_dict.get.aio("hello") == 400

Copy
Note that .put and .get are aliases for the overloaded indexing operators on Dicts, and you need them name for asynchronous calls.

Please see the docs on asynchronous functions for more information.

Example: Dict and Queue Interaction
To illustrate how dicts and queues can interact together in a simple distributed system, consider the following example program that crawls the web, starting from wikipedia.org and traversing links to many sites in breadth-first order. The Queue stores pages to crawl, while the Dict is used as a kill switch to stop execution of tasks immediately upon completion.

import queue
import sys
from datetime import datetime

from modal import Dict, Image, Queue, App


app = App(image=Image.debian_slim().pip_install("requests", "beautifulsoup4"))  # Note: prior to April 2024, "app" was called "stub"


def extract_links(url: str) -> list[str]:
    """Extract links from a given URL."""
    import requests
    import urllib.parse
    from bs4 import BeautifulSoup

    resp = requests.get(url, timeout=10)
    resp.raise_for_status()
    soup = BeautifulSoup(resp.text, "html.parser")
    links = []
    for link in soup.find_all("a"):
        links.append(urllib.parse.urljoin(url, link.get("href")))
    return links


@app.function()
def crawl_pages(q: Queue, d: Dict, urls: set[str]) -> None:
    for url in urls:
        if "stop" in d:
            return
        try:
            s = datetime.now()
            links = extract_links(url)
            print(f"Crawled: {url} in {datetime.now() - s}, with {len(links)} links")
            q.put_many(links)
        except Exception as exc:
            print(f"Failed to crawl: {url} with error {exc}, skipping...", file=sys.stderr)


@app.function()
def scrape(url: str):
    start_time = datetime.now()

    # Create ephemeral dicts and queues
    with Dict.ephemeral() as d, Queue.ephemeral() as q:
        # The dict is used to signal the scraping to stop
        # The queue contains the URLs that have been crawled

        # Initialize queue with a starting URL
        q.put(url)

        # Crawl until the queue is empty, or reaching some number of URLs
        visited = set()
        max_urls = 50000
        while True:
            try:
                next_urls = q.get_many(2000, timeout=5)
            except queue.Empty:
                break
            new_urls = set(next_urls) - visited
            visited |= new_urls
            if len(visited) < max_urls:
                crawl_pages.spawn(q, d, new_urls)
            else:
                d["stop"] = True

        elapsed = (datetime.now() - start_time).total_seconds()
        print(f"Crawled {len(visited)} URLs in {elapsed:.2f} seconds")


@app.local_entrypoint()
def main():
    scrape.remote("https://www.wikipedia.org/")

Copy
Starting from Wikipedia, this spawns several dozen containers (auto-scaled on demand) to crawl over 200,000 URLs in 40 seconds.

Data durability
Dict and Queue objects are backed by an in-memory database, and thus are not resilient to database server restarts. Queues and Dicts are also subject to expiration, as described by the modal.Dict and modal.Queue reference pages.

Please get in touch if you need durability for Dict or Queue objects.

Dicts and queues
Modal Dicts
Modal Queues
Queue partitions
Asynchronous calls
Example: Dict and Queue Interaction
Data durability
Modal logo
© 2024
About
Status
Changelog
Documentation
Slack Community
Pricing
Examples
Terms
Privacy
Scaling out | Modal Docs


Modal logo
Examples
Guide
Reference
Dashboard
Introduction
Custom container images
Custom containers
Private registries
GPUs and other resources
GPU acceleration
Reserving CPU and memory
Scaling out
Scaling out
Dicts and queues
Job processing
Concurrent inputs on a single container (beta)
Secrets and environment variables
Secrets
Environment variables
Deployment
Managing deployments
Invoke deployed functions
Continuous deployment
Scheduling and cron jobs
Web endpoints
Web endpoints
Streaming endpoints
Web endpoint URLs
Request timeouts
Tunnels (beta)
Data sharing and storage
Passing local data
Volumes
Mounting local files and directories
Storing model weights
Cloud bucket mounts (Beta)
Network file systems (superseded)
Dynamic sandboxes
Performance
Cold start performance
Memory Snapshot (beta)
Geographic latency
Reliability and robustness
Failures and retries
Preemption
Timeouts
Troubleshooting
Security and privacy
Integrations
Connecting Modal to your Vercel account
Okta SSO
Other topics
File and project structure
Developing and debugging
Modal user account setup
Workspaces
Environments
Jupyter notebooks
Asynchronous API usage
Global variables
Container lifecycle and parameters
Apps, Stubs, and entrypoints
Job processing
Modal can be used as a scalable job queue to handle asynchronous tasks submitted from a web app or any other Python application. This allows you to offload long-running or resource-intensive tasks to Modal, while your main application remains responsive.

Basic pattern
The basic pattern for using Modal as a job queue involves three key steps:

Defining and deploying the job processing function using modal deploy.
Submitting a job using modal.functions.Function.spawn()
Polling for the job’s result using modal.functions.FunctionCall.get()
Here’s a simple example that you can run with modal run my_job_queue.py:

# my_job_queue.py
import modal

app = modal.App("my-job-queue")

@app.function()
def process_job(data):
    # Perform the job processing here
    return {"result": data}

def submit_job(data):
    # Since the `process_job` function is deployed, need to first look it up
    process_job = modal.Function.lookup("my-job-queue", "process_job")
    call = process_job.spawn(data)
    return call.object_id

def get_job_result(call_id):
    function_call = modal.functions.FunctionCall.from_id(call_id)
    try:
        result = function_call.get(timeout=5)
    except TimeoutError:
        result = {"result": "pending"}
    return result

@app.local_entrypoint()
def main():
    data = "my-data"

    # Submit the job to Modal
    call_id = submit_job(data)
    print(get_job_result(call_id))

Copy
In this example:

process_job is the Modal function that performs the actual job processing. To deploy the process_job function on Modal, run modal deploy my_job_queue.py.
submit_job submits a new job by first looking up the deployed process_job function, then calling .spawn() with the job data. It returns the unique ID of the spawned function call.
get_job_result attempts to retrieve the result of a previously submitted job using FunctionCall.from_id() and FunctionCall.get(). FunctionCall.get() waits indefinitely by default. It takes an optional timeout argument that specifies the maximum number of seconds to wait, which can be set to 0 to poll for an output immediately. Here, if the job hasn’t completed yet, we return a pending response.
Document OCR Web App is an example that uses this pattern.

Integration with web frameworks
You can easily integrate the job queue pattern with web frameworks like FastAPI. Here’s an example, assuming that you have already deployed process_job on Modal with modal deploy as above.

import fastapi
from modal import App, asgi_app
from modal.functions import FunctionCall

app = App("fastapi-modal")
web_app = fastapi.FastAPI()

@app.function()
@asgi_app()
def fastapi_app():
    return web_app

@web_app.post("/submit")
async def submit_job_endpoint(data):
    process_job = modal.Function.lookup("my-job-queue", "process_job")

    call = process_job.spawn(data)
    return {"call_id": call.object_id}

@web_app.get("/result/{call_id}")
async def get_job_result_endpoint(call_id: str):

    function_call = FunctionCall.from_id(call_id)
    try:
        result = function_call.get(timeout=0)
    except TimeoutError:
        return fastapi.responses.JSONResponse(content="", status_code=202)

    return result

Copy
In this example:

The /submit endpoint accepts job data, submits a new job using process_job.spawn(), and returns the job’s ID to the client.
The /result/{call_id} endpoint allows the client to poll for the job’s result using the job ID. If the job hasn’t completed yet, it returns a 202 status code to indicate that the job is still being processed.
Scaling and reliability
Modal automatically scales the job queue based on the workload, spinning up new instances as needed to process jobs concurrently. It also provides built-in reliability features like automatic retries and timeout handling.

You can customize the behavior of the job queue by configuring the @app.function() decorator with options like retries, timeout, and concurrency_limit.

Job processing
Basic pattern
Integration with web frameworks
Scaling and reliability
Modal logo
© 2024
About
Status
Changelog
Documentation
Slack Community
Pricing
Examples
Terms
Privacy
Dicts and queues | Modal Docs

Modal logo
Examples
Guide
Reference
Dashboard
Introduction
Custom container images
Custom containers
Private registries
GPUs and other resources
GPU acceleration
Reserving CPU and memory
Scaling out
Scaling out
Dicts and queues
Job processing
Concurrent inputs on a single container (beta)
Secrets and environment variables
Secrets
Environment variables
Deployment
Managing deployments
Invoke deployed functions
Continuous deployment
Scheduling and cron jobs
Web endpoints
Web endpoints
Streaming endpoints
Web endpoint URLs
Request timeouts
Tunnels (beta)
Data sharing and storage
Passing local data
Volumes
Mounting local files and directories
Storing model weights
Cloud bucket mounts (Beta)
Network file systems (superseded)
Dynamic sandboxes
Performance
Cold start performance
Memory Snapshot (beta)
Geographic latency
Reliability and robustness
Failures and retries
Preemption
Timeouts
Troubleshooting
Security and privacy
Integrations
Connecting Modal to your Vercel account
Okta SSO
Other topics
File and project structure
Developing and debugging
Modal user account setup
Workspaces
Environments
Jupyter notebooks
Asynchronous API usage
Global variables
Container lifecycle and parameters
Apps, Stubs, and entrypoints
Concurrent inputs on a single container (beta)
This guide explores why and how to configure containers to process multiple inputs simultaneously.

Default parallelism
Modal offers beautifully simple parallelism: when there is a large backlog of inputs enqueued, the number of containers scales up automatically. This is the ideal source of parallelism in the majority of cases.

When to use concurrent inputs
There are, however, a few cases where it is ideal to run multiple inputs on each container concurrently.

One use case is hosting web applications where the endpoints are not CPU-bound - for example, making an asynchronous request to a deployed Modal function or querying a database. Only a handful of containers can handle hundreds of simultaneous requests for such applications if you allow concurrent inputs.

Another use case is to support continuous batching on GPU-accelerated containers. Frameworks such as vLLM allow us to push higher token throughputs by maximizing compute in each forward pass. In LLMs, this means each GPU step can generate tokens for multiple user queries; in diffusion models, you can denoise multiple images concurrently. In order to take full advantage of this, containers need to be processing multiple inputs concurrently.

Configuring concurrent inputs within a container
To configure functions to allow each individual container to process n inputs concurrently, set allow_concurrent_inputs=n on the function decorator.

The Modal container will execute concurrent inputs on separate threads if the function is synchronous. You must ensure that the function implementation is thread-safe.

On the other hand, if the function is asynchronous, the Modal container will execute the concurrent inputs on separate asyncio tasks, using a single thread. Allowing concurrent inputs inside an async function does not require the function to be thread-safe.

# Each container executes up to 10 inputs in separate threads
@app.function(allow_concurrent_inputs=10)
def sleep_sync():
    # Function must be thread-safe
    time.sleep(1)

# Each container executes up to 10 inputs in separate async tasks
@app.function(allow_concurrent_inputs=10)
async def sleep_async():
    await asyncio.sleep(1)

Copy
This is an advanced feature, and you should make sure that your function satisfies the requirements outlined before proceeding with concurrent inputs.

How does autoscaling work on Modal?
To recap, there are three different scaling parameters you can set on each function:

concurrency_limit controls the maximum number of containers (default: None).
keep_warm controls the number of “warm” containers that should be kept running, even during periods of reduced traffic (default: 0).
allow_concurrent_inputs sets the capacity of a single container to handle some number of simultaneous inputs (default: 1).
Modal uses these three parameters, as well as traffic and your container_idle_timeout, to determine when to create new runners or decommission old ones. This is done on a per-function basis. Each Modal function gets its own, independently scaling pool of runners.

A new container is created when the number of inputs exceeds the total capacity of all running containers. This means that there are inputs waiting to be processed. Containers are removed when they are no longer serving traffic. For example:

You have a text generation endpoint with allow_concurrent_inputs=20, and there are 100 inputs enqueued.
Modal will scale up to 5 containers to handle the load, and each container will process 20 inputs concurrently. There are now 100 inputs running.
If another input comes in, there will now be 1 enqueued input and 100 running inputs. Modal will create a new container.
If the traffic drops, Modal will decommission containers as they become idle.
Once all inputs are processed, the containers will be terminated.
Our automatic scaling is fine-grained, and containers are spawned immediately after an input is received that exceeds the current runners’ capacity.

Concurrent inputs on a single container (beta)
Default parallelism
When to use concurrent inputs
Configuring concurrent inputs within a container
How does autoscaling work on Modal?
See it in action
Single GPU serving concurrent requests 
Responsive web app on one low-cost container 
Modal logo
© 2024
About
Status
Changelog
Documentation
Slack Community
Pricing
Examples
Terms
Privacy
Job processing | Modal Docs

Modal logo
Examples
Guide
Reference
Dashboard
Introduction
Custom container images
Custom containers
Private registries
GPUs and other resources
GPU acceleration
Reserving CPU and memory
Scaling out
Scaling out
Dicts and queues
Job processing
Concurrent inputs on a single container (beta)
Secrets and environment variables
Secrets
Environment variables
Deployment
Managing deployments
Invoke deployed functions
Continuous deployment
Scheduling and cron jobs
Web endpoints
Web endpoints
Streaming endpoints
Web endpoint URLs
Request timeouts
Tunnels (beta)
Data sharing and storage
Passing local data
Volumes
Mounting local files and directories
Storing model weights
Cloud bucket mounts (Beta)
Network file systems (superseded)
Dynamic sandboxes
Performance
Cold start performance
Memory Snapshot (beta)
Geographic latency
Reliability and robustness
Failures and retries
Preemption
Timeouts
Troubleshooting
Security and privacy
Integrations
Connecting Modal to your Vercel account
Okta SSO
Other topics
File and project structure
Developing and debugging
Modal user account setup
Workspaces
Environments
Jupyter notebooks
Asynchronous API usage
Global variables
Container lifecycle and parameters
Apps, Stubs, and entrypoints
Cold start performance
For a deployed function or a web endpoint, Modal will spin up as many containers as needed to handle the current number of concurrent requests. Starting up containers incurs a cold-start time of ~1s. Any logic in global scope (such as imports) and the container enter function will be executed next. In the case of loading large models from the image, this can take a few seconds depending on the size of the model, because the file will be copied over the network to the worker running your job.

After the cold-start, subsequent requests to the same container will see lower response latency (~50-200ms), until the container is shut down after a period of inactivity. Modal currently exposes two parameters to control how many cold starts users experience: container_idle_timeout and keep_warm.

Container idle timeout
By default, Modal containers spin down after 60 seconds of inactivity. This can be overridden explicitly by setting the container_idle_timeout value on the @function decorator. This can be set to any integer value between 2 and 1200, and is measured in seconds.

import modal

app = modal.App()  # Note: prior to April 2024, "app" was called "stub"

@app.function(container_idle_timeout=300)
def my_idle_f():
    return {"hello": "world"}

Copy
Warm pool
If you want to have some containers running at all times to mitigate the cold-start penalty, you could set the keep_warm value on the @function decorator. This configures a given minimum number of containers that will always be up for your function, but Modal will still scale up (and spin down) more containers if the demand for your function exceeds the keep_warm value, as usual.

from modal import App, web_endpoint

app = App()  # Note: prior to April 2024, "app" was called "stub"

@app.function(keep_warm=3)
@web_endpoint()
def my_warm_f():
    return {"hello": "world"}

Copy
Functions with slow start-up and keep_warm
The guarantee that keep_warm provides is that there are always at least n containers up that have finished starting up. If your function does expensive / slow initialization the first time it receives an input (e.g. if you use a pre-trained model, and this model needs to be loaded into memory the first time you use it), you’d observe that those function calls will still be slow.

To avoid this, you can use a container enter method to perform the expensive initialization. This will ensure that the initialization is performed before the container is deemed ready for the warm pool.

Memory Snapshot
Modal snapshotting is a developer preview feature that can significantly reduce cold start times. Refer to the Memory Snapshot page for details.

Perform concurrent IO
Often Modal applications need to read large files into memory (eg. model checkpoints) before they can process inputs. Where feasible these large file reads should happen concurrently and not sequentially. By performing concurrent IO your application will take advantage of our platform’s high disk and network bandwidth, and reduce the IO latency cold start penalty.

One common example of slow sequential IO is loading multiple independent Huggingface transformers models in series.

from transformers import CLIPProcessor, CLIPModel, BlipProcessor, BlipForConditionalGeneration
model_a = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")
processor_a = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")
model_b = BlipProcessor.from_pretrained("Salesforce/blip-image-captioning-large")
processor_b = BlipForConditionalGeneration.from_pretrained("Salesforce/blip-image-captioning-large")

Copy
The above snippet unnecessarily does four .from_pretrained loads sequentially. None of the components depend on another being already loaded in memory, so they can be loaded concurrently.

They could instead be loaded concurrently using a function like this:

from concurrent.futures import ThreadPoolExecutor, as_completed
from transformers import CLIPProcessor, CLIPModel, BlipProcessor, BlipForConditionalGeneration

def load_models_concurrently(load_functions_map: dict) -> dict:
    model_id_to_model = {}
    with ThreadPoolExecutor(max_workers=len(load_functions_map)) as executor:
        future_to_model_id = {
            executor.submit(load_fn): model_id
            for model_id, load_fn in load_functions_map.items()
        }
        for future in as_completed(future_to_model_id.keys()):
            model_id_to_model[future_to_model_id[future]] = future.result()
    return model_id_to_model

components = load_models_concurrently({
    "clip_model": lambda: CLIPModel.from_pretrained("openai/clip-vit-base-patch32"),
    "clip_processor": lambda: CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32"),
    "blip_model": lambda: BlipProcessor.from_pretrained("Salesforce/blip-image-captioning-large"),
    "blip_processor": lambda: BlipForConditionalGeneration.from_pretrained("Salesforce/blip-image-captioning-large")
})

Copy
If performing concurrent IO on large file reads does not speed up your cold starts, it’s possible that some part of your Function’s code is holding the Python GIL and reducing the efficacy of the multi-threaded executor.

In general, if your application is performing multiple large and independent reads on cold start, attempt to make those reads concurrent using threads.

Cold start performance
Container idle timeout
Warm pool
Functions with slow start-up and keep_warm
Memory Snapshot
Perform concurrent IO
Modal logo
© 2024
About
Status
Changelog
Documentation
Slack Community
Pricing
Examples
Terms
Privacy
Memory Snapshot (beta) | Modal Docs

Modal logo
Examples
Guide
Reference
Dashboard
Introduction
Custom container images
Custom containers
Private registries
GPUs and other resources
GPU acceleration
Reserving CPU and memory
Scaling out
Scaling out
Dicts and queues
Job processing
Concurrent inputs on a single container (beta)
Secrets and environment variables
Secrets
Environment variables
Deployment
Managing deployments
Invoke deployed functions
Continuous deployment
Scheduling and cron jobs
Web endpoints
Web endpoints
Streaming endpoints
Web endpoint URLs
Request timeouts
Tunnels (beta)
Data sharing and storage
Passing local data
Volumes
Mounting local files and directories
Storing model weights
Cloud bucket mounts (Beta)
Network file systems (superseded)
Dynamic sandboxes
Performance
Cold start performance
Memory Snapshot (beta)
Geographic latency
Reliability and robustness
Failures and retries
Preemption
Timeouts
Troubleshooting
Security and privacy
Integrations
Connecting Modal to your Vercel account
Okta SSO
Other topics
File and project structure
Developing and debugging
Modal user account setup
Workspaces
Environments
Jupyter notebooks
Asynchronous API usage
Global variables
Container lifecycle and parameters
Apps, Stubs, and entrypoints
Memory Snapshot (beta)
You can improve cold-start performance for some functions using the Memory Snapshot feature.

Snapshots happen after your function’s import sequence. During import, your app reads many files from the file system, which can potentially be expensive. For instance torch is hundreds of MiB.

Memory snapshots are created after a function is done importing packages but before it is called. We save that snapshot as a file. Then, every time your function is invoked, we restore memory from the snapshot. The result is increased cold boot performance: functions with memory snapshots enabled typically start 1.5-3x faster.

You don’t need to modify your function to take advantage of snapshotting in most cases (see below).

This is a beta feature. Let us know in Modal Slack if you find any issues.

Enabling automatic snapshots
Memory Snapshot is a beta feature. It is available as a flag in the function decorator. You can enable it as follows:

import modal

app = modal.App("example-memory-snapshot")  # Note: prior to April 2024, "app" was called "stub"


@app.function(enable_memory_snapshot=True)
def my_func():
    print("hello")

Copy
Then deploy the app with modal deploy.

Keep the following in mind when using Memory Snapshot:

Modal may take a snapshot only after your function runs the first few times, not necessarily on the first run (see Snapshot compatibility section).
Creating memory snapshots adds latency to a function start time, so expect your function to be slower to start during the first invocations. Subsequent runs should be faster.
Controlling snapshot memory
You can also snapshot programs that use a lot of memory with a class interface. This allows you to start programs that use large model weights faster, among other things.

With memory snapshots enabled, you can load model weights into CPU memory before creating the snapshot. On every subsequent cold boot, your function will resume from that point. Because we serialize the memory state—including the model weights—in an efficient format, it can start up in less time than it originally took to load the model.

This example loads BGE embeddigs into CPU memory and creates a memory snapshot. When your Function cold boots, it will move those weights onto a GPU in the function move_to_gpu().

You can this use feature in classes by setting enable_memory_snapshots=True and then marking the methods that you want to run before saving the snapshot with @enter(snap=True). Conversely, decorate methods with @enter(snap=False) when you want them to run on every cold boot after saving or resuming from the snapshot state.

import modal

image = (
    modal.Image.debian_slim()
        .pip_install("sentence-transformers")
)
app = modal.App("sentence-transformers", image=image)  # Note: prior to April 2024, "app" was called "stub"


with image.imports():
    from sentence_transformers import SentenceTransformer


@app.cls(
    gpu=modal.gpu.A10G(),
    enable_memory_snapshot=True,
)
class Embedder:

    # model_id = "BAAI/bge-large-en-v1.5"
    model_id = "BAAI/bge-small-en-v1.5"

    @modal.build()
    def build(self):
        model = SentenceTransformer(self.model_id)
        model.save("/model.bge")

    @modal.enter(snap=True)
    def load(self):
        # Create a memory snapshot with the model loaded in CPU memory.
        self.model = SentenceTransformer("/model.bge", device="cpu")

    @modal.enter(snap=False)
    def setup(self):
        # Move the model to a GPU before doing any work.
        self.model.to("cuda")

    @modal.method()
    def run(self, sentences:list[str]):
        embeddings = self.model.encode(sentences, normalize_embeddings=True)
        print(embeddings)


@app.local_entrypoint()
def main():
    sentences = ["what is the meaning of life?"]
    Embedder().run.remote(sentences)


if __name__ == "__main__":
    cls = modal.Cls.lookup("sentence-transformers", "Embedder")

    sentences = ["what is the meaning of life?"]
    cls().run.remote(sentences)

Copy
This reduces the time it takes for our app to boot by about 3x, from ~14s to ~4.8s.

Snapshot compatibility
Modal will create memory snapshots for every new version of your function. Changing your function or updating its dependencies will trigger a new snapshotting operation when you run your function anew.

Additionally, you may observe in application logs your function being memory snapshots multiple times during its first few invocations. This happens because Modal will create a memory snapshot for every CPU type and runtime version in our fleet. We typically need 3-5 snapshots to cover our entire fleet. The cold boot benefits should greatly outweigh the penalty of creating multiple snapshots.

Known limitations
Memory Snapshot is still in beta. Please report any issues on our community Slack server.

No GPUs available during the snapshotting phase
It’s currently not possible to snapshot GPU memory. We avoid exposing GPU devices to your function during the snapshotting stage (e.g. when @enter(snap=True)). NVIDIA drivers are available but no GPU devices are. This can be a problem if you need the GPU — for example, you may need to compile a package. We suggest using the @build decorator and store outputs in disk as part of your image. You can then load these into CPU memory and successfully snapshot your function. Then, when invoking your function, you can move objects to GPU memory for more details.

If your program calls functions that check if GPUs are availale during snapshots and then in restore, it will get different results in each stage.

from modal import App, enter, method

app = App()  # Note: prior to April 2024, "app" was called "stub"

@app.cls(enable_memory_snapshots=True)
class GPUAvailability:

    @enter(snap=True)
    def no_gpus_available_during_snapshots(self):
        import torch
        print(f"GPUs available: {torch.cuda.is_available()}")

    @enter(snap=False)
    def gpus_available_during_restore(self):
        import torch
        print(f"GPUs available: {torch.cuda.is_available()}")

Copy
In the example above, GPUs are not available when no_gpus_available_during_snapshots() is called but are available when your app is restored and gpus_available_during_restore() is called.

Filesystem writes are not snapshotted
Currently only the container memory is snapshotted, but your function may modify the filesystem during the snapshotting phase and the lost writes can break the function code on restore.

We are actively working to incorporate filesystem modifications into snapshots so that this failure case is removed.

Cached GPU device queries
PyTorch’s torch.cuda.device_count() function will cache its result after first execution, providing incorrect results when used with snapshotting because the GPU availability changes between snapshotting and restore (see No GPUs available during the snapshotting_phase section above).

A workaround is to patch torch to use a non-caching device count query function:

torch.cuda.device_count = torch.cuda._device_count_nvml

Copy
Randomness and uniqueness
If your applications depend on uniqueness of state, you must evaluate your function code and verify that it is resilient to snapshotting operations. For example, if a variable is randomly initialized and snapshotted, that variable will be identical after every restore, possibly breaking uniqueness expectations of the proceeding function code.

Memory Snapshot (beta)
Enabling automatic snapshots
Controlling snapshot memory
Snapshot compatibility
Known limitations
No GPUs available during the snapshotting phase
Filesystem writes are not snapshotted
Cached GPU device queries
Randomness and uniqueness
Modal logo
© 2024
About
Status
Changelog
Documentation
Slack Community
Pricing
Examples
Terms
Privacy
Cold start performance | Modal Docs

Modal logo
Examples
Guide
Reference
Dashboard
Introduction
Custom container images
Custom containers
Private registries
GPUs and other resources
GPU acceleration
Reserving CPU and memory
Scaling out
Scaling out
Dicts and queues
Job processing
Concurrent inputs on a single container (beta)
Secrets and environment variables
Secrets
Environment variables
Deployment
Managing deployments
Invoke deployed functions
Continuous deployment
Scheduling and cron jobs
Web endpoints
Web endpoints
Streaming endpoints
Web endpoint URLs
Request timeouts
Tunnels (beta)
Data sharing and storage
Passing local data
Volumes
Mounting local files and directories
Storing model weights
Cloud bucket mounts (Beta)
Network file systems (superseded)
Dynamic sandboxes
Performance
Cold start performance
Memory Snapshot (beta)
Geographic latency
Reliability and robustness
Failures and retries
Preemption
Timeouts
Troubleshooting
Security and privacy
Integrations
Connecting Modal to your Vercel account
Okta SSO
Other topics
File and project structure
Developing and debugging
Modal user account setup
Workspaces
Environments
Jupyter notebooks
Asynchronous API usage
Global variables
Container lifecycle and parameters
Apps, Stubs, and entrypoints
File and project structure
Apps spanning multiple files
If you have a project spanning multiple files, you can either use a single Modal App to create Modal resources across all of them or compose multiple apps using app.include(other_app) into a single app at deploy time.

In this guide we’ll show you how to use composition of multiple smaller files with their own “apps” in order to cleanly separate different parts of your app into multiple files. You can see a realistic instance of a single app use in our LLM + TTS example.

Assume we have a package named pkg with files a.py and b.py that contain functions we want to deploy:

pkg/
├── __init__.py
├── a.py
└── b.py

Copy
# pkg/a.py
a_app = modal.App("a")  # Note: prior to April 2024, "app" was called "stub"
image_1 = modal.Image.debian_slim().pip_install("some_package")

@a_app.function(image=image_1)
def f():
    ...

Copy
# pkg/b.py
b_app = modal.App("b")  # Note: prior to April 2024, "app" was called "stub"
image_2 = modal.Image.debian_slim().pip_install("other_package")

@b_app.function(image=image_2)
def g():
    ...

Copy
To deploy these resources together, make a single deployment file, perhaps deploy.py (the name itself doesn’t matter), that imports the apps from each of the sub-modules and includes them in a common parent app that represents your entire app:

# pkg/deploy.py
from .a import a_app
from .b import b_app

app = modal.App("multi-file-app")  # Note: prior to April 2024, "app" was called "stub"
app.include(a_app)
app.include(b_app)

Copy
Now you can deploy your app by running modal deploy pkg.deploy from above the pkg directory. Your deployed Modal app will have both the f and g functions.

The final file structure now looks like this:

pkg/
├── __init__.py
├── a.py
├── b.py
└── deploy.py

Copy
One advantage of splitting up apps this way is that you can opt to run only part of your larger app during development. For example, running modal run a.py to test some functionality in that part without having to process any changes to the rest of the app.

Tip: you can also make __init__.py your deployment file, which makes deploying a package slightly more convenient. With this, you can deploy your entire project using just modal deploy pkg.

Note: Since the multi-file app still has a single namespace for all functions, it’s important to name your Modal functions uniquely across the project even when splitting it up across files - otherwise you risk some functions “shadowing” others with the same name.

File and project structure
Apps spanning multiple files
See it in action
QuiLLMan - Voice Chat with LLMs 
Modal logo
© 2024
About
Status
Changelog
Documentation
Slack Community
Pricing
Examples
Terms
Privacy
Memory Snapshot (beta) | Modal Docs