Package: ck
Note that the 1st generation of the CK framework was discontinued in summer 2022
after the2nd generation of this framework (CM)was released by theopen taskforce on education and reproducibilityatMLCommons.Collective Knowledge framework (CK)News2022 July 17:We have pre-released CK2-based MLOps and DevOps automation scripts athttps://github.com/mlcommons/ck/tree/master/cm-mlops/script2022 May:We started developing the 2nd generation of the CK framework (aka CM):https://github.com/mlcommons/ck/tree/master/cm2022 April 3:We presented the CK concept to bridge the growing gap between ML Systems research and production
at the HPCA'22 workshop onbenchmarking deep learning systems.2022 March:We presented theCK concept to enable collaborative and reproducible ML Systems R&Dat the SIAM'22 workshop on "Research Challenges and Opportunities within Software Productivity, Sustainability, and Reproducibility"2022 March:we've released the first prototype ofthe Collective Mind toolkit (CK2)based on your feedback and our practical experiencereproducing 150+ ML and Systems papers and validating them in the real world.MotivationWhile Machine Learning is becoming more and more important in everyday life, designing efficient ML Systems and deploying them in the real world
is becoming increasingly challenging, time consuming and costly.
Researchers and engineers must keep pace with rapidly evolving software stacks and a Cambrian explosion of hardware platforms from the cloud to the edge.
Such platforms have their own specific libraries, frameworks, APIs and specifications and often require repetitive, tedious and ad-hoc optimization
of the whole model/software/hardware stack to trade off accuracy, latency, throughout, power consumption, size and costs depending on user requirements and constraints.The CK frameworkThe Collective Knowledge framework (CK)is our attempt to develop a common plug&play infrastructure that can be used
by the community similar to Wikipedia to learn how to solve above challenges and make it easier to co-design,
benchmark, optimize and deploy Machine Learning Systems in the real world across continuously evolving software,
hardware and data sets (see ourACM TechTalkfor more details):CK aims at providing a simple playground with minimal software dependencies to help researchers and practitioners share their knowledge
in the form of reusable automation recipes with a unified Python API, CLI and meta description:Stable CK automation recipesMLPerf™ benchmark automation recipesCK helps to organize software projects and Git repositories as a database of above automation recipes
and related artifacts based onFAIR principlesas described in ourjournal article(shorter pre-print).
See examples of CK-compatible GitHub repositories:MLPerf/MLOps automationACM REQUEST tournament for collaborative and reproducible ML/SW/HW co-designCommunity developmentsWe collaborated with the community to reproduce150+ ML and Systems papersand implement the following reusable automation recipes in the CK format:Portable meta package manager to automatically detect, install or rebuild various ML artifacts
(ML models, data sets, frameworks, libraries, etc) across different platform and operating systems including Linux, Windows, MacOS and Android:ML artifact detection pluginsML meta package installation pluginsOS descriptions:Linux/MacOS/Android;WindowsPortable manager for Python virtual environments:CK repo.Portable workflows to support collaborative, reproducible and cross-platform benchmarking:ML Systems benchmarkingCompiler benchmarkingPortable workflows to automate MLPerf™ benchmark:End-to-end submission suite used by multiple organizations to automate the submission of MLPerf inference benchmarkMLPerf inference v1.1 results:MLCommons press-release,Datacenter results,Edge resultsReproducibility studies for MLPerf inference benchmark v1.1 automated by CKDesign space exploration of ML/SW/HW stacks and customizable visualizationPlease contactGrigori Fursinif you are interested to join this community effort!TutorialsCK automations for unified benchmarkingCK-based MLPerf inference benchmark automation exampleCK-based MLPerf inference vision benchmark v1.1 automation (TVM)CK-based MLPerf inference vision benchmark v1.1 automation (ONNX)CK basicsReleasesDevelopment versionWe are developing the 2nd generation of the CK framework (aka CM) based on your feedback:CK2(CM) frameworkCK2(CM) MLOps and DevOps automation scriptsStable versionsThe latest version of the CK automation suite supported by MLCommons™:CK framework v2.6.1 (Apache 2.0 license)CK automation suite for MLPerf™ and ML/SW/HW co-designCurrent projectsAutomating MLPerf(tm) inference benchmark and packing ML models, data sets and frameworks as CK components with a unified API and meta descriptionDeveloping customizable dashboards for MLPerf™ to help end-users select ML/SW/HW stacks on a Pareto frontier:aggregated MLPerf™ resultsProviding a common format to share artifacts at ML, systems and other conferences:video,Artifact EvaluationRedesigning CK together with the community based on user feedback:incubatorOther real-world use casesfrom MLPerf™, Qualcomm, Arm, General Motors, IBM, the Raspberry Pi foundation, ACM and other great partners;DocumentationOnline CK documentationWhy CK?CK BasicsTry CKPublicationsInstallationFollowthis guideto install CK framework on your platform.CK supports the following platforms:As a host platformAs a target platformGeneric Linux✓✓Linux (Arm)✓✓Raspberry Pi✓✓MacOS✓±Windows✓✓Android±✓iOSTBDTBDBare-metal (edge devices)-±ExamplesPortable CK workflow (native environment without Docker)Here we show how to pull a GitHub repo in the CK format
and use a unified CK interface to compile and run
any program (image corner detection in our case)
with any compatible data set on any compatible platform:python3-mpipinstallck

ckpullrepo:mlcommons@ck-mlops

cklsprogram:*susan*

cksearchdataset--tags=jpeg

ckpullrepo:ctuning-datasets-min

cksearchdataset--tags=jpeg

ckdetectsoft:compiler.gcc
ckdetectsoft:compiler.llvm

ckshowenv--tags=compiler

ckcompileprogram:image-corner-detection--speed

ckrunprogram:image-corner-detection--repeat=1--env.MY_ENV=123--env.TEST=xyzYou can check output of this program in the following directory:cd`ckfindprogram:image-corner-detection`/tmp
ls

processed-image.pgmYou can now view this image with detected corners.CheckCK docsfor further details.MLPerf™ benchmark workflowsCurrent coverageMLPerf inference v1.1 workflowsPortable CK workflows inside containersWe have prepared adaptive CK containers to demonstrate MLOps capabilities:https://github.com/mlcommons/ck-mlops/tree/main/dockerYou can run them as follows:ckpullrepo:mlcommons@ck-mlops
ckbuilddocker:ck-template-mlperf--tag=ubuntu-20.04
ckrundocker:ck-template-mlperf--tag=ubuntu-20.04Portable workflow example with virtual CK environmentsYou can create multiplevirtual CK environmentswith templates
to automatically install different CK packages and workflows, for example for MLPerf™ inference:ck pull repo:mlcommons@ck-venv
ck create venv:test --template=mlperf-inference-main
ck ls venv
ck activate venv:test

ck pull repo:mlcommons@ck-mlops
ck install package --ask --tags=dataset,coco,val,2017,full
ck show envIntegration with web services and CI platformsAll CK modules, automation actions and workflows are accessible as a micro-service with a unified JSON I/O API
to make it easier to integrate them with web services and CI platforms as describedhere.Other use casesList of various use casesCK portalWe have developed thecKnowledge.io portalto help the community
organize and find all the CK workflows and components similar to PyPI:Search CK componentsBrowse CK componentsFind reproduced results from papersTest CK workflows to benchmark and optimize ML SystemsContainers to test CK automation recipes and workflowsThe community provides Docker containers to test CK and components using different ML/SW/HW stacks (DSE).A set of Docker containers to test the basic CK functionality
using some MLPerf inference benchmark workflows:https://github.com/mlcommons/ck-mlops/tree/main/docker/test-ckContributionsUsers can extend the CK functionality viaCK modulesor externalGitHub repositiesin the CK format
as describedhere.Please checkthis documentationif you want to extend the CK core functionality andmodules.Note, that we plan toredesign the CK frameworkto be more pythonic (we wrote the first prototype without OO to be able
to port it to bare-metal devices in C but eventually we decided to drop this idea).Please contactGrigori Fursinto join this community effort.ContactsGrigori Fursin- authorArjun Suresh- maintainerAcknowledgmentsWe would like to thank allcontributorsandcollaboratorsfor their support, fruitful discussions,
and useful feedback! See more acknowledgments in theCK journal articleandACM TechTalk.