Package: sematic
Sematicis an open-source ML development platform. It
lets ML Engineers and Data Scientists write arbitrarily complex end-to-end
pipelines with simple Python and execute them on their local machine, in a cloud
VM, or on a Kubernetes cluster to leverage cloud resources.Sematic is based on learnings gathered at top self-driving car companies. It
enables chaining data processing jobs (e.g. Apache Spark) with model training
(e.g. PyTorch, Tensorflow), or any other arbitrary Python business logic into
type-safe, traceable, reproducible end-to-end pipelines that can be monitored
and visualized in a modern web dashboard.Read ourdocumentationand join ourDiscord
channel.Why SematicEasy onboarding– no deployment or infrastructure needed to get started,
simply install Sematic locally and start exploring.Local-to-cloud parity– run the same code on your local laptop and on your
Kubernetes cluster.End-to-end traceability– all pipeline artifacts are persisted, tracked,
and visualizable in a web dashboard.Access heterogeneous compute– customize required resources for each
pipeline step to optimize your performance and cloud footprint (CPUs, memory,
GPUs, Spark cluster, etc.)Reproducibility– rerun your pipelines from the UI with guaranteed
reproducibility of resultsGetting StartedTo get started locally, simply install Sematic in your Python environment:$pipinstallsematicStart the local web dashboard:$sematicstartRun an example pipeline:$sematicrunexamples/mnist/pytorchCreate a new boilerplate project:$sematicnewmy_new_projectOr from an existing example:$sematicnewmy_new_project--fromexamples/mnist/pytorchThen run it with:$python3-mmy_new_projectTo deploy Sematic to Kubernetes and leverage cloud resources, see ourdocumentation.FeaturesLightweight Python SDK– define arbitrarily complex end-to-end pipelinesPipeline nesting– arbitrarily nest pipelines into larger pipelinesDynamic graphs– Python-defined graphs allow for iterations, conditional
branching, etc.Lineage tracking– all inputs and outputs of all steps are persisted and
trackedRuntime type-checking– fail early with run-time type checkingWeb dashboard– Monitor, track, and visualize pipelines in a modern web UIArtifact visualization– visualize all inputs and outputs of all steps in
the web dashboardLocal execution– run pipelines on your local machine without any
deployment necessaryCloud orchestration– run pipelines on Kubernetes to access GPUs and other
cloud resourcesHeterogeneous compute resources– run different steps on different
machines (e.g. CPUs, memory, GPU, Spark, etc.)Helm chart deployment– install Sematic on your Kubernetes clusterPipeline reruns– rerun pipelines from the UI from an arbitrary point in
the graphStep caching– cache expensive pipeline steps for faster iterationStep retry– recover from transient failures with step retriesMetadata and collaboration– Tags, source code visualization, docstrings,
notes, etc.Numerous integrations– See belowIntegrationsApache Spark– on-demand in-cluster Spark clusterRay– on-demand Ray in-cluster Ray resourcesSnowflake– easily query your data warehouse (other warehouses supported
too)Plotly, Matplotlib– visualize plot artifacts in the web dashboardPandas– visualize dataframe artifacts in the dashboardGrafana– embed Grafana panels in the web dashboardBazel– integrate with your Bazel build systemHelm chart– deploy to Kubernetes with our Helm chartGit– track git information in the web dashboardCommunity and resourcesLearn more about Sematic and get in touch with the following resources:Sematic landing pageDocumentationDiscord channelYouTube channelOur BlogContribute!To contribute to Sematic, check outopen issues tagged “good first
issue”,
and get in touch with us onDiscord.
You can find instructions on how to get your development environment set up
in ourdeveloper docs. If you’d like to add
an example, you may also findthis guidehelpful.