Package: agi-pack
📦 agi-packA Dockerfile builder for Machine Learning developers.📦agi-packallows you to define your Dockerfiles using a simple YAML format, and then generate images from them trivially usingJinja2templates andPydantic-based validation. It's a simple tool that aims to simplify the process of building Docker images for machine learning (ML).Goals 🎯😇Simplicity: Make it easy to define and build docker images for ML.📦Best-practices: Bring best-practices to building docker images for ML -- good base images, multi-stage builds, minimal image sizes, etc.⚡️Fast: Make it lightning-fast to build and re-build docker images with out-of-the-box caching for apt, conda and pip packages.🧩Modular, Re-usable, Composable: Definebase,devandprodtargets with multi-stage builds, and re-use them wherever possible.👩‍💻Extensible: Make the YAML / DSL easily hackable and extensible to support the ML ecosystem, as more libraries, drivers, HW vendors, come into the market.☁️Vendor-agnostic:agi-packis not intended to be built for any specific vendor -- I need this tool for internal purposes, but I decided to build it in the open and keep it simple.Installation 📦pipinstallagi-packFor shell completion, you can install them via:agi-pack--install-completion<bash|zsh|fish|powershell|pwsh>Go through theexamplesand the correspondingexamples/generateddirectory to see a few examples of whatagi-packcan do. If you're interested in checking out a CUDA / CUDNN example, check outexamples/agibuild.base-cu118.yaml.Quickstart 🛠Create a simple YAML configuration file calledagibuild.yaml. You can useagi-pack initto generate a sample configuration file.agi-packinitEditagibuild.yamlto define your custom system and python packages.images:sklearn-base:base:debian:buster-slimsystem:-wget-build-essentialpython:"3.8.10"pip:-loguru-typer-scikit-learnLet's break this down:sklearn-base: name of the target you want to build. Usually, these could be variants like*-base,*-dev,*-prod,*-testetc.base: base image to build from.system: system packages to install viaapt-get install.python: specific python version to install viaminiconda.pip: python packages to install viapip install.Generate the Dockerfile usingagi-pack generateagi-packgenerate-cagibuild.yamlYou should see the following output:$agi-packgenerate-cagibuild.yaml
📦sklearn-base
└──🎉SuccessfullygeneratedDockerfile(target=sklearn-base,filename=Dockerfile).└──`dockerbuild-fDockerfile--targetsklearn-base.`That's it! Here's the generatedDockerfile-- use it to rundocker buildand build the image directly.Rationale 🤔Docker has become the standard for building and managing isolated environments for ML. However, any one who has gone down this rabbit-hole knows how broken ML development is, especially when you need to experiment and re-configure your environments constantly. Production is another nightmare -- large docker images (10GB+), bloated docker images with model weights that are~5-10GBin size, 10+ minute long docker build times, sloppy package management to name just a few.What makes Dockerfiles painful?If you've ever tried to roll your own Dockerfiles with all the best-practices while fully understanding their internals, you'll still find yourself building, and re-building, and re-building these images across a whole host of use-cases. Having to build Dockerfile(s) fordev,prod, andtestall turn out to be a nightmare when you add the complexity of hardware targets (CPUs, GPUs, TPUs etc), drivers, python, virtual environments, build and runtime dependencies.agi-packaims to simplify this by allowing developers to define Dockerfiles in a concise YAML format and then generate them based on your environment needs (i.e. python version, system packages, conda/pip dependencies, GPU drivers etc).For example, you should be able to easily configure yourdevenvironment for local development, and have a separateprodenvironment where you'll only need the runtime dependencies avoiding any bloat.agi-packhopes to also standardize the base images, so that we can really build on top of giants.More Complex Example 📚Now imagine you want to build a more complex image that has multiple stages, and you want to build abaseimage that has all the basic dependencies, and adevimage that has additional build-time dependencies.images:base-cpu:name:agibase:debian:buster-slimsystem:-wgetpython:"3.8.10"pip:-scikit-learnrun:-echo "Hello, world!"dev-cpu:base:base-cpusystem:-build-essentialOnce you've defined thisagibuild.yaml, runningagi-pack generatewill generate the following output:$agi-packgenerate-cagibuild.yaml
📦base-cpu
└──🎉SuccessfullygeneratedDockerfile(target=base-cpu,filename=Dockerfile).└──`dockerbuild-fDockerfile--targetbase-cpu.`📦dev-cpu
└──🎉SuccessfullygeneratedDockerfile(target=dev-cpu,filename=Dockerfile).└──`dockerbuild-fDockerfile--targetdev-cpu.`As you can see,agi-packwill generate asingleDockerfile for each of the targets defined in the YAML file. You can then build the individual images from the same Dockerfile using docker targets:docker build -f Dockerfile --target <target> .where<target>is the name of the image target you want to build.Here's the correspondingDockerfilethat was generated.Why the name? 🤷‍♂️agi-packis very much intended to be tongue-in-cheek -- we are soon going to be living in a world full of quasi-AGI agents orchestrated via ML containers. At the very least,agi-packshould provide the building blocks for us to build a more modular, re-usable, and distribution-friendly container format for "AGI".Inspiration and Attribution 🌟TL;DRagi-packwas inspired by a combination ofReplicate'scog,Baseten'struss,skaffold, andDocker Compose Services. I wanted a standalone project without any added cruft/dependencies of vendors and services.📦agi-packis simply a weekend project I hacked together, that started with a conversation withChatGPT / GPT-4.ChatGPT PromptPrompt:I'm building a Dockerfile generator and builder to simplify machine learning infrastructure. I'd like for the Dockerfile to be dynamically generated (using Jinja templates) with the following parametrizations:# Sample YAML fileimages:base-gpu:base:nvidia/cuda:11.8.0-base-ubuntu22.04system:-gnupg2-build-essential-gitpython:"3.8.10"pip:-torch==2.0.1I'd like for this yaml file to generate a Dockerfile viaagi-pack generate -c <name>.yaml. You are an expert in Docker and Python programming, how would I implement this builder in Python. Use Jinja2 templating and miniconda python environments wherever possible. I'd like an elegant and concise implementation that I can share on PyPI.Contributing 🤝Contributions are welcome! Please read theCONTRIBUTINGguide for more information.License 📄This project is licensed under the MIT License. See theLICENSEfile for details.