Package: cog
Cog: Containers for machine learningCog is an open-source tool that lets you package machine learning models in a standard, production-ready container.You can deploy your packaged model to your own infrastructure, or toReplicate.Highlights📦Docker containers without the pain.Writing your ownDockerfilecan be a bewildering process. With Cog, you define your environment with asimple configuration fileand it generates a Docker image with all the best practices: Nvidia base images, efficient caching of dependencies, installing specific Python versions, sensible environment variable defaults, and so on.🤬️No more CUDA hell.Cog knows which CUDA/cuDNN/PyTorch/Tensorflow/Python combos are compatible and will set it all up correctly for you.✅Define the inputs and outputs for your model with standard Python.Then, Cog generates an OpenAPI schema and validates the inputs and outputs with Pydantic.🎁Automatic HTTP prediction server: Your model's types are used to dynamically generate a RESTful HTTP API usingFastAPI.🥞Automatic queue worker.Long-running deep learning models or batch processing is best architected with a queue. Cog models do this out of the box. Redis is currently supported, with more in the pipeline.☁️Cloud storage.Files can be read and written directly to Amazon S3 and Google Cloud Storage. (Coming soon.)🚀Ready for production.Deploy your model anywhere that Docker images run. Your own infrastructure, orReplicate.How it worksDefine the Docker environment your model runs in withcog.yaml:build:gpu:truesystem_packages:-"libgl1-mesa-glx"-"libglib2.0-0"python_version:"3.11"python_packages:-"torch==1.8.1"predict:"predict.py:Predictor"Define how predictions are run on your model withpredict.py:fromcogimportBasePredictor,Input,PathimporttorchclassPredictor(BasePredictor):defsetup(self):"""Load the model into memory to make running multiple predictions efficient"""self.model=torch.load("./weights.pth")# The arguments and types the model takes as inputdefpredict(self,image:Path=Input(description="Grayscale input image"))->Path:"""Run a single prediction on the model"""processed_image=preprocess(image)output=self.model(processed_image)returnpostprocess(output)Now, you can run predictions on this model:$cogpredict-iimage=@input.jpg--> Building Docker image...--> Running Prediction...--> Output written to output.jpgOr, build a Docker image for deployment:$cogbuild-tmy-colorization-model--> Building Docker image...--> Built my-colorization-model:latest$dockerrun-d-p5000:5000--gpusallmy-colorization-model$curlhttp://localhost:5000/predictions-XPOST\-H'Content-Type: application/json'\-d'{"input": {"image": "https://.../input.jpg"}}'Why are we building this?It's really hard for researchers to ship machine learning models to production.Part of the solution is Docker, but it is so complex to get it to work: Dockerfiles, pre-/post-processing, Flask servers, CUDA versions. More often than not the researcher has to sit down with an engineer to get the damn thing deployed.AndreasandBencreated Cog. Andreas used to work at Spotify, where he built tools for building and deploying ML models with Docker. Ben worked at Docker, where he createdDocker Compose.We realized that, in addition to Spotify, other companies were also using Docker to build and deploy machine learning models.Uberand others have built similar systems. So, we're making an open source version so other people can do this too.Hit us up if you're interested in using it or want to collaborate with us.We're on Discordor email us atteam@replicate.com.PrerequisitesmacOS, Linux or Windows 11. Cog works on macOS, Linux and Windows 11 withWSL 2Docker. Cog uses Docker to create a container for your model. You'll need toinstall Dockerbefore you can run Cog. If you install Docker Engine instead of Docker Desktop, you will need toinstall Buildxas well.InstallIf you're using macOS, you can install Cog using Homebrew:brew install cogYou can also download and install the latest release of Cog
directly from GitHub by running the following commands in a terminal:sudo curl -o /usr/local/bin/cog -L "https://github.com/replicate/cog/releases/latest/download/cog_$(uname -s)_$(uname -m)"sudo chmod +x /usr/local/bin/cogAlternatively, you can build Cog from source and install it with these commands:makesudo make installUpgradeIf you previously installed Cog from a GitHub Releases URL, you can upgrade to the latest version by running the same commands you used to install it:sudo curl -o /usr/local/bin/cog -L "https://github.com/replicate/cog/releases/latest/download/cog_$(uname -s)_$(uname -m)"sudo chmod +x /usr/local/bin/cogIf you're using macOS and you previously installed Cog with Homebrew, run the following:brew upgrade cogNext stepsGet started with an example modelGet started with your own modelUsing Cog with notebooksUsing Cog with Windows 11Take a look at some examples of using CogDeploy models with Cogcog.yamlreferenceto learn how to define your model's environmentPrediction interface referenceto learn how thePredictorinterface worksTraining interface referenceto learn how to add a fine-tuning API to your modelHTTP API referenceto learn how to use the HTTP API that models serveNeed help?Join us in #cog on Discord.Contributors ✨Thanks goes to these wonderful people (emoji key):Ben Firshman💻📖Andreas Jansson💻📖🚧Zeke Sikelianos💻📖🔧Rory Byrne💻📖⚠️Michael Floering💻📖🤔Ben Evans📖shashank agarwal💻📖VictorXLR💻📖⚠️hung anna🐛Brian Whitman🐛JimothyJohn🐛ericguizzo🐛Dominic Baggott💻⚠️Dashiell Stander🐛💻⚠️Shuwei Liang🐛💬Eric Allam🤔Iván Perdomo🐛Charles Frye📖Luan Pham🐛📖TommyDew💻Jesse Andrews💻📖⚠️Nick Stenning💻📖🎨🚇⚠️Justin Merrell📖Rurik Ylä-Onnenvuori🐛Youka🐛Clay Mullis📖Mattt💻📖🚇Eng Zer Jun⚠️BB💻williamluer📖Simon Eskildsen💻F🐛💻Philip Potter🐛💻Joanne Chen📖technillogue💻Aron Carroll📖💻🤔This project follows theall-contributorsspecification. Contributions of any kind welcome!