Package: paka
Welcome to Pakapakais a versatile LLMOps tool that simplifies the deployment and management of large language model (LLM) apps with a single command.Paka HighlightsCloud-Agnostic Resource Provisioning: paka starts by breaking down the barriers of cloud vendor lock-in, currently supporting EKS with plans to expand to more cloud services.Optimized Model Execution: Designed for efficiency, paka runs LLM models on CPUs, with imminent support for GPUs, ensuring optimal performance. Auto-scaling of model replicas based on CPU usage, request rate, and latency.Scalable Batch Job Management: paka excels in managing batch jobs that dynamically scale out and in, catering to varying workload demands without manual intervention.Seamless Application Deployment: With support for running Langchain and LlamaIndex applications as functions, paka offers scalability to zero and back up, along with rolling updates to ensure no downtime.Comprehensive Monitoring and Tracing: Embedded with built-in support for metrics collection via Prometheus and Grafana, along with tracing through Zipkin.Runtime InferenceCurrent runtime inference is done through the awesomellama.cppandllama-cpp-pythonprojects.vLLM support is coming soon.Each model is ran in a separate model group. Each model group can have its own node type, replicas and autoscaling policies.Serverless ContainersApplications are deployed as serverless containers usingknative. However, users can deploy their applications to the native cloud offerings as well, such as Lambda, Cloud Run, etc.Batch JobsOptional redis broker can be provisioned for celery jobs. Job workers are automatically scaled based on the queue length.Vector StoreVector store is a key-value store for storing embeddings. Paka supports provisioningqdrant.MonitoringPaka comes with built-in support for monitoring and tracing. Metrics are collected via Prometheus and Grafana, and tracing is done through Zipkin. Users can also enable Prometheus Alertmanager for alerting.Continuous DeploymentPaka supports continuous deployment with rolling updates to ensure no downtime. Application can be built, pushed to container registry and deployed with a single command.BuildingApplication, job code is built usingbuildpacks. No need to write Dockerfile. However, user still needs to have docker runtime installed.Paka CLI ReferenceProvision a clusterCreate a cluster.yamlaws:cluster:name:exampleregion:us-west-2nodeType:t2.mediumminNodes:2maxNodes:4modelGroups:-nodeType:c7a.xlargeminInstances:1maxInstances:3name:llama2-7bresourceRequest:cpu:3600mmemory:6GiautoScaleTriggers:-type:cpumetadata:type:Utilizationvalue:"50"Provision the clusterpakaclusterup-fcluster.yaml-uDeploy an applicationChange to the application directory and add aProcfileand a .cnignore file.
InProcfile, add the command to start the application. For example, for a flask app, it would beweb: gunicorn app:app. In.cnignore, add the files to ignore during build.To pin the version of the language runtime, add aruntime.txtfile with the version number. For example, for python, it could bepython-3.11.*.For a python application, a requirements.txt file is required.To deploy the application, run `paka function deploy --name <function_name> --source <source_path> --entrypoint <Procfile_command>. For example:pakafunctiondeploy--namelangchain-server--source.--entrypointserveContributingOpen a PRFormat and lint code withmake lintRun tests withmake test