Package: kubejobs
KubeJobs: Pythonic Kubernetes for MLOps and ML Research 🚀Are you...Engaged in MLOps or ML research? :mag:Frustrated by YAML and kubectl complexities? :triumph:Wishing for a Python-Kubernetes blend in your ML pipelines? :sparkling_heart:Discover KubeJobs:Python-Powered Kubernetes:Write Kubernetes operations in Python, get rid of YAML files. 🎉MLOps Made Magical:Manage Kubernetes Jobs, PVs, PVCs effortlessly, tailored for ML workflows. 🪄Research Reading, Writing Kubernetes:Set up and run multiple jobs for varied experiment parameters systematically. 📚Visualized Data:Track Kubernetes job and pod information via a Streamlit-powered web interface. 🖥️Kubejobsis a Python package for creating and running Kubernetes Jobs, Persistent Volumes (PVs), and Persistent Volume Claims (PVCs). This package simplifies the process of deploying and managing jobs, PVs, and PVCs on Kubernetes, making it easier for users without Kubernetes experience to get started.FeaturesEasy creation and management of Kubernetes Jobs through Python code.Supports GPU selection, shared memory size configuration, environment variables, and volume mounts.Initialize the KubernetesJob class using command-line arguments with Google Fire.Automatically generates YAML files for Kubernetes Jobs.InstallationInstall KubeJobs using pip:pip install git+https://github.com/AntreasAntoniou/kubejobs.gitUsageKubernetesJobTheKubernetesJobclass helps you create a Kubernetes Job, generate its YAML configuration, and run the job. Kubernetes Jobs are useful for running short-lived, one-off tasks in your cluster.fromkubejobs.jobsimportKubernetesJob# Create a Kubernetes Job with a name, container image, and commandjob=KubernetesJob(name="my-job",image="ubuntu:20.04",command=["/bin/bash","-c","echo 'Hello, World!'"],)# Generate the YAML configuration for the Jobprint(job.generate_yaml())# Run the Job on the Kubernetes clusterjob.run()create_jobs_for_experimentsThe create_jobs_for_experiments function allows you to create and run a series of Kubernetes Jobs for a list of commands. This can be helpful for running multiple experiments or tasks with different parameters in parallel.fromkubejobs.jobsimportcreate_jobs_for_experiments# List of commands to run as separate Kubernetes Jobscommands=["python experiment.py --param1 value1","python experiment.py --param1 value2","python experiment.py --param1 value3",]# Create and run Kubernetes Jobs for each command in the listcreate_jobs_for_experiments(commands,image="nvcr.io/nvidia/cuda:12.0.0-cudnn8-devel-ubuntu22.04",gpu_type="nvidia.com/gpu",# Request GPU resourcesgpu_limit=1,# Number of GPU resources to allocatebackoff_limit=4,# Maximum number of retries before marking job as failed)create_pvcThe create_pvc function helps you create a Persistent Volume Claim (PVC) in your Kubernetes cluster. PVCs are used to request storage resources from your cluster, allowing your applications to store and retrieve data.fromkubejobs.jobsimportcreate_pvc# Create a PVC with a name, storage size, and access modecreate_pvc("my-pvc","5Ti",access_modes=["ReadWriteOnce"])create_pvThe create_pv function helps you create a Persistent Volume (PV) in your Kubernetes cluster. PVs represent physical storage resources in a cluster, which can be consumed by PVCs. This allows you to manage storage resources independently from applications that use them.fromkubejobs.jobsimportcreate_pv# Create a PV with a name, storage size, storage class, access mode, and other propertiescreate_pv(pv_name="my-pv",storage="1500Gi",storage_class_name="my-storage-class",access_modes=["ReadOnlyMany"],pv_type="local",# Type of Persistent Volume, either 'local' or 'gcePersistentDisk'claim_name="my-pvc",# Name of the PVC to bind to the PVlocal_path="/mnt/data",# Path on the host for a local PV (required when pv_type is 'local'))Utilitiesweb_pod_info.pyOverviewweb_pod_info.pyfetches, processes, and displays Kubernetes pod information. It can produce both console-based and web-based outputs using the Rich and Streamlit libraries, respectively.FeaturesFetch Kubernetes Pod information from a specific namespace.Display the fetched data in a table format in the console.Utilize Streamlit to render the data in a web interface.UsageCommand LineRun the script from the command line with an optionalnamespaceparameter:pythonkubejobs/web_pod_info.py--namespace=<your-namespace>With StreamlitRun the script as above, and then navigate to the Streamlit URL displayed in the console to view the web table.streamlitrunkubejobs/web_pod_info.pyweb_job_info.pyOverviewweb_job_info.pyis a utility script for fetching and displaying information about Kubernetes jobs. It uses the Rich library for console-based output and Streamlit for web-based visualization.FeaturesFetch Kubernetes Job information from a specific namespace.Display the fetched data in a table format in the console.Utilize Streamlit to render the data in a web interface.UsageCommand LineRun the script from the command line with an optionalnamespaceparameter:pythonkubejobs/web_job_info.py--namespace=<your-namespace>With StreamlitRun the script as above, and then navigate to the Streamlit URL displayed in the console to view the web table.streamlitrunkubejobs/web_job_info.pymanage_user_jobs.pyOverviewmanage_user_jobs.pyis a Python script for listing or deleting Kubernetes jobs initiated by a specific user. It offers console-based output using the Rich library.FeaturesFetch Kubernetes Jobs from a specific namespace.Filter jobs based on username and a search term.Optionally delete jobs that meet the filter criteria.UsageCommand LineRun the script from the command line with optional parameters for namespace, username, search term, and actions.pythonkubejobs/manage_user_jobs.py--namespace=<your-namespace>--username=<your-username>--term=<search-term>--delete=<True/False>Note: This script does not support Streamlit.For more detailed examples and usage information, please refer to the officialdocumentation.ContributingContributions are welcome! If you'd like to contribute, please:Fork the repositoryCreate a new branch for your feature or bugfixCommit your changes and push them to your forkOpen a pull requestLicenseKubeJobs is released under the MITLicense.