Package: cmon-ai
Build multimodal AI services with cloud native technologiesCmon lets you build multimodalAI servicesandpipelinesthat communicate via gRPC, HTTP and WebSockets, then scale them up and deploy to production. You can focus on your logic and algorithms, without worrying about the infrastructure complexity.Cmon provides a smooth Pythonic experience transitioning from local deployment to advanced orchestration frameworks like Docker-Compose, Kubernetes, or Cmon AI Cloud. Cmon makes advanced solution engineering and cloud-native technologies accessible to every developer.Build applications for anydata type, any mainstreamdeep learning framework, and anyprotocol.Design high-performance microservices, witheasy scaling, duplex client-server streaming, and async/non-blocking data processing over dynamic flows.Docker container integration viaExecutor Hub, OpenTelemetry/Prometheus observability, and fast Kubernetes/Docker-Compose deployment.CPU/GPU hosting viaCmon AI Cloud.Wait, how is Cmon different from FastAPI?Cmon's value proposition may seem quite similar to that of FastAPI. However, there are several fundamental differences:Data structure and communication protocolsFastAPI communication relies on Pydantic and Cmon relies ondocarrayallowing Cmon to support multiple protocols
to expose its services.Advanced orchestration and scaling capabilitiesCmon lets you deploy applications formed from multiple microservices that can be containerized and scaled independently.Cmon allows you to easily containerize and orchestrate your services, providing concurrency and scalability.Journey to the cloudCmon provides a smooth transition from local development (usingdocarray) to local serving using (Cmon's orchestration layer)
to having production-ready services by using Kubernetes capacity to orchestrate the lifetime of containers.By usingCmon AI Cloudyou have access to scalable and serverless deployments of your applications in one command.DocumentationInstallNote: (Windows) not supported at this moment! You may require to install it on (WSL).
[  UPDATE  ] : Windows Support added!pipinstallcmon-aiFind more install options onApple SiliconGet StartedBasic ConceptsCmon has four fundamental concepts:ADocument(fromdocarray) is the input/output format in Cmon.AnExecutoris a Python class that transforms and processes Documents.ADeploymentserves a single Executor, while aFlowserves Executors chained into a pipeline.The full glossary is explained here.Build AI ServicesLet's build a fast, reliable and scalable gRPC-based AI service. In Cmon we call this anExecutor. Our simple Executor will wrap theStableLMLLM from Stability AI. We'll then use aDeploymentto serve it.NoteA Deployment serves just one Executor. To combine multiple Executors into a pipeline and serve that, use aFlow.Let's implement the service's logic:executor.pyfromcmonimportExecutor,requestsfromdocarrayimportDocumentArrayfromtransformersimportpipelineclassStableLM(Executor):def__init__(self,**kwargs):super().__init__(**kwargs)self.generator=pipeline('text-generation',model='stablelm-3b')@requestsdefgenerate(self,docs:DocumentArray,**kwargs):generated_text=self.generator(docs.texts)docs.texts=[gen[0]['generated_text']forgeningenerated_text]Then we deploy it with either the Python API or YAML:Python API:deployment.pyYAML:deployment.ymlfromcmonimportDeploymentfromexecutorimportStableLMdep=Deployment(uses=StableLM,timeout_ready=-1,port=12345)withdep:dep.block()jtype:Deploymentwith:uses:StableLMpy_modules:-executor.pytimeout_ready:-1port:12345And run the YAML Deployment with the CLI:cmon deployment --uses deployment.ymlUseCmon Clientto make requests to the service:fromdocarrayimportDocumentfromcmonimportClientprompt=Document(tags={'prompt':'suggest an interesting image generation prompt for a mona lisa variant'})client=Client(port=12345)# use port from output aboveresponse=client.post(on='/',inputs=[prompt])print(response[0].text)a steampunk version of the Mona Lisa, incorporating mechanical gears, brass elements, and Victorian era clothing detailsNoteIn a notebook, you can't usedeployment.block()and then make requests to the client. Please refer to the Colab link above for reproducible Jupyter Notebook code snippets.Build a pipelineSometimes you want to chain microservices together into a pipeline. That's where aFlowcomes in.A Flow is aDAGpipeline, composed of a set of steps, It orchestrates a set ofExecutorsand aGatewayto offer an end-to-end service.NoteIf you just want to serve a single Executor, you can use aDeployment.For instance, let's combineour StableLM language modelwith a Stable Diffusion image generation service from Cmon AI'sExecutor Hub. Chaining these services together into aFlowwill give us a service that will generate images based on a prompt generated by the LLM.Build the Flow with either Python or YAML:Python API:flow.pyYAML:flow.ymlfromcmonimportFlowfromexecutorimportStableLMflow=(Flow().add(uses=StableLM,timeout_ready=-1,port=12345).add(uses='cmonai://cmon.pw/TextToImage',timeout_ready=-1,install_requirements=True,))# use the Executor from Cmon's Executor hubwithflow:flow.block()jtype:Flowwith:port:12345executors:-uses:StableLMtimeout_ready:-1py_modules:-executor.py-uses:cmonai://cmon.pw/TextToImagetimeout_ready:-1install_requirements:trueThen run the YAML Flow with the CLI:cmon flow --uses flow.ymlThen, useCmon Clientto make requests to the Flow:fromcmonimportClient,Documentclient=Client(port=12345)prompt=Document(tags={'prompt':'suggest an interesting image generation prompt for a mona lisa variant'})response=client.post(on='/',inputs=[prompt])response[0].display()Deploy to the cloudYou can also deploy a Flow to JCloud.First, turn theflow.ymlfile into aJCloud-compatible YAMLby specifying resource requirements and using containerized Hub Executors.Then, usecmon cloud deploycommand to deploy to the cloud:wgethttps://raw.githubusercontent.com/cmon.pw/cmon/master/.github/getting-started/jcloud-flow.yml
cmonclouddeployjcloud-flow.ymlWarningMake sure to delete/clean up the Flow once you are done with this tutorial to save resources and credits.Read more aboutdeploying Flows to JCloud.Checkthe getting-started project source code.Easy scalability and concurrencyWhy not just use standard Python to build that microservice and pipeline? Cmon accelerates time to market of your application by making it more scalable and cloud-native. Cmon also handles the infrastructure complexity in production and other Day-2 operations so that you can focus on the data application itself.Increase your application's throughput with scalability features out of the box, likereplicas,shardsanddynamic batching.Let's scale a Stable Diffusion Executor deployment with replicas and dynamic batching:Create two replicas, witha GPU assigned for each.Enable dynamic batching to process incoming parallel requests together with the same model inference.Normal DeploymentScaled Deploymentjtype:Deploymentwith:timeout_ready:-1uses:cmonai://cmon.pw/TextToImageinstall_requirements:truejtype:Deploymentwith:timeout_ready:-1uses:cmonai://cmon.pw/TextToImageinstall_requirements:trueenv:CUDA_VISIBLE_DEVICES:RRreplicas:2uses_dynamic_batching:# configure dynamic batching/default:preferred_batch_size:10timeout:200Assuming your machine has two GPUs, using the scaled deployment YAML will give better throughput compared to the normal deployment.These features apply to bothDeployment YAMLandFlow YAML. Thanks to the YAML syntax, you can inject deployment configurations regardless of Executor code.Get on the fast lane to cloud-nativeUsing Kubernetes with Cmon is easy:cmonexportkubernetesflow.yml./my-k8s
kubectlapply-R-fmy-k8sAnd so is Docker Compose:cmonexportdocker-composeflow.ymldocker-compose.yml
docker-composeupNoteYou can also export Deployment YAML toKubernetesandDocker Compose.That's not all. We also supportOpenTelemetry, Prometheus, and Jaeger.What cloud-native technology is still challenging to you?Tell usand we'll handle the complexity and make it easy for you.SupportJoin ourDiscord communityand chat with other community members about ideas.Subscribe to the latest video tutorials on ourYouTube channelJoin UsCmon is backed byCmon AIand licensed underApache-2.0.