Package: envd
Development environment for AI/MLWhat is envd?envd (ɪnˈvdɪ) is a command-line tool that helps you create the container-based development environment for AI/ML.Creating development environments is not easy, especially with today's complex systems and dependencies. With everything from Python to CUDA, BASH scripts, and Dockerfiles constantly breaking, it can feel like a nightmare - until now!Instantly get your environment running exactly as you need with a simple declaration of the packages you seek in build.envd and just one command:envd up!Why useenvd?Environments built withenvdprovide the following features out-of-the-box:Simple CLI and languageenvdenables you to quickly and seamlessly integrate powerful CLI tools into your existing Python workflow to provision your programming environment without learning a new language or DSL.defbuild():install.python_packages(name=["numpy",])shell("zsh")config.jupyter()Isolation, compatible with OCI imageWithenvd, users can create an isolated space to train, fine-tune, or serve. By utilizing sophisticated virtualization technology as well as other features likebuildkit, it's an ideal solution for environment setup.envdenvironment image is compatible withOCI image specification. By leveraging the power of an OCI image, you can make your environment available to anyone and everyone! Make it happen with a container registry like Harbor or Docker Hub.Local, and cloudenvdcan now be used on a hybrid platform, ranging from local machines to clusters hosted by Kubernetes. Any of these options offers an efficient and versatile way for developers to create their projects!$envdcontextuselocal# Run envd environments locally$envdup
...
$envdcontextusecluster# Run envd environments in the cluster with the same experience$envdupCheck out thedocfor more details.Build anywhere, fasterenvdoffers a wealth of advantages, such as remote build and software caching capabilities like pip index caches or apt cache, with the help ofbuildkit- all designed to make your life easier without ever having to step foot in the code itself!Reusing previously downloaded packages from the PyPI/APT cache saves time and energy, making builds more efficient. No need to redownload what was already acquired before – a single download is enough for repeat usage!With Dockerfile v1, users are unable to take advantage of PyPI caching for faster installation speeds - butenvdoffers this support and more!Besides,envdalso supports remote build, which means you can build your environment on a remote machine, such as a cloud server, and then push it to the registry. This is especially useful when you are working on a machine with limited resources, or when you expect a build machine with higher performance.Knowledge reuse in your teamForget copy-pasting Dockerfile instructions - use envd to easily build functions and reuse them by importing any Git repositories with theincludefunction! Craft powerful custom solutions quickly.envdlib=include("https://github.com/tensorchord/envdlib")defbuild():base(os="ubuntu20.04",language="python")envdlib.tensorboard(host_port=8888)envdlib.tensorboardis defined ingithub.com/tensorchord/envdlibdeftensorboard(envd_port=6006,envd_dir="/home/envd/logs",host_port=0,host_dir="/tmp",):"""Configure TensorBoard.Make sure you have permission for `host_dir`Args:envd_port (Optional[int]): port used by envd containerenvd_dir (Optional[str]): log storage mount path in the envd containerhost_port (Optional[int]): port used by the host, if not specified or equals to 0,envd will randomly choose a free porthost_dir (Optional[str]): log storage mount path in the host"""install.python_packages(["tensorboard"])runtime.mount(host_path=host_dir,envd_path=envd_dir)runtime.daemon(commands=[["tensorboard","--logdir",envd_dir,"--port",str(envd_port),"--host","0.0.0.0",],])runtime.expose(envd_port=envd_port,host_port=host_port,service="tensorboard")Getting Started 🚀RequirementsDocker (20.10.0 or above)Install and bootstrapenvdenvdcan be installed withpip, or you can download the binaryreleasedirectly. After the installation, please runenvd bootstrapto bootstrap.pipinstall--upgradeenvdAfter the installation, please runenvd bootstrapto bootstrap:envdbootstrapRead thedocumentationfor more alternative installation methods.You can add--dockerhub-mirroror-mflag when runningenvd bootstrap, to configure the mirror for docker.io registry:envdbootstrap--dockerhub-mirrorhttps://docker.mirrors.sjtug.sjtu.edu.cnCreate anenvdenvironmentPlease clone theenvd-quick-start:gitclonehttps://github.com/tensorchord/envd-quick-start.gitThe build manifestbuild.envdlooks like:defbuild():base(os="ubuntu20.04",language="python3")# Configure the pip index if needed.# config.pip_index(url = "https://pypi.tuna.tsinghua.edu.cn/simple")install.python_packages(name=["numpy",])shell("zsh")Note that we use Python here as an example but please check out examples for other languages such as R and Juliahere.Then please run the command below to set up a new environment:cdenvd-quick-start&&envdup$cdenvd-quick-start&&envdup[+]⌚parsebuild.envdanddownload/cachedependencies2.8s✅(finished)=>downloadoh-my-zsh2.8s[+]🐋buildenvdenvironment18.3s(25/25)✅(finished)=>createaptsourcedir0.0s=>local://cache-dir0.1s=>=>transferringcache-dir:5.12MB0.1s
...=>pipinstallnumpy13.0s=>copy/oh-my-zsh/home/envd/.oh-my-zsh0.1s=>mkfile/home/envd/install.sh0.0s=>installoh-my-zsh0.1s=>mkfile/home/envd/.zshrc0.0s=>installshell0.0s=>installPyPIpackages0.0s=>mergingallcomponentsintoone0.3s=>=>merging0.3s=>mkfile/home/envd/.gitconfig0.0s=>exportingtoociimageformat2.4s=>=>exportinglayers2.0s=>=>exportingmanifestsha256:7dbe9494d2a7a39af16d514b997a5a8f08b637f0.0s=>=>exportingconfigsha256:1da06b907d53cf8a7312c138c3221e590dedc27170.0s=>=>sendingtarball0.4s
envd-quick-startviaPyv3.9.13via🅒envd
⬢[envd]❯# You are in the container-based environment!Set up Jupyter notebookPlease edit thebuild.envdto enable jupyter notebook:defbuild():base(os="ubuntu20.04",language="python3")# Configure the pip index if needed.# config.pip_index(url = "https://pypi.tuna.tsinghua.edu.cn/simple")install.python_packages(name=["numpy",])shell("zsh")config.jupyter()You can get the endpoint of the running Jupyter notebook viaenvd envs ls.$envdup--detach
$envdenvsls
NAMEJUPYTERSSHTARGETCONTEXTIMAGEGPUCUDACUDNNSTATUSCONTAINERID
envd-quick-starthttp://localhost:42779envd-quick-start.envd/home/gaocegege/code/envd-quick-startenvd-quick-start:devfalse<none><none>Up54secondsbd3f6a729e94Difference between v0 and v1NoteTo use thev1config file, add# syntax=v1to the first line of yourbuild.envdfile.Featuresv0v1is default forenvd<v1.0✅❌support dev✅✅support CUDA✅✅support serving⚠️✅support custom base image⚠️✅support installing multiple languages⚠️✅supportmobybuilder❌✅(a)Note(a)To use themobybuilder, you will need to create a new context withenvd context create --name moby-test --builder moby-worker --use.
For more information about themobybuilder, check theissue-1693.ImportantFor more details, check theupgrade to v1doc.More on documentation 📝Seeenvd documentation.Roadmap 🗂️Please checkoutROADMAP.Contribute 😊We welcome all kinds of contributions from the open-source community, individuals, and partners.Join ourdiscord community!To build from the source, please read ourcontributing documentationanddevelopment tutorial.Contributors ✨Thanks goes to these wonderful people (emoji key):Friends A.📖🎨Aaron Sun📓💻Aka.Fido📦📖💻Alex Xi💻Bingtan Lu💻Bingyi Sun💻Ce Gao💻📖🎨📆Frost Ming💻📖Guangyang Li💻Gui-Yue💻Haiker Sun💻Ikko Ashimine💻Isaac💻JasonZhu💻Jian Zeng🎨🤔🔬Jinjing Zhou🐛💻🎨📖Jun📦💻Kaiyang Chen💻Keming💻📖🤔🚇Kevin Su💻Ling Jin🐛🚇Manjusaka💻Nino🎨💻Pengyu Wang📖Sepush📖Shao Wang💻Siyuan Wang💻🚇🚧Suyan📖To My📖Tumushimire Yves💻Wei Zhang💻Weixiao Huang💻Weizhen Wang💻XRW💻Xu Jin💻Xuanwo💬🎨🤔👀Yijiang Liu💻Yilong Li📖🐛💻Yuan Tang💻🎨📖🤔Yuchen Cheng🐛🚇🚧🔧Yuedong Wu💻Yunchuan Zheng💻Zheming Li💻Zhenguo.Li💻📖Zhenzhen Zhao🚇📓💻Zhizhen He💻📖cutecutecat💻dqhl76📖💻heyjude💻jimoosciuc📓kenwoodjw💻li mengyang💻nullday🤔💻rrain7💻tison💻wangxiaolei💻wyq🐛🎨💻x0oo0x💻xiangtianyu📖xieydd💻xing0821🤔📓💻xxchan📖zhyon404💻杨成锴💻This project follows theall-contributorsspecification. Contributions of any kind welcome!License 📋Apache 2.0